Re: [Python-Dev] My work on Python3 and non-ascii paths is done

[Paul Moore]

On 19 October 2010 05:52, Éric Araujo  wrote:
> Congratulations Victor!  This is not a small feat.  The PSU should send
> you cookies to thank you, but they won’t since they don’t exist and

What? Cookies don't exist???

Paul.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] About resolution “accepted ” on the tracker

[Antoine Pitrou]

On Mon, 18 Oct 2010 22:27:06 -0500
Ron Adam  wrote:
> 
> Could something like (or parts of) the following work?  It would have 
> assignment and module keywords items as well.

Well, this is very nice, except that the more complicated the form is,
the less likely people are to fill it (and even less, fill it properly).

+1 for relabelling "unit test needed" as simply "test needed".

Regards

Antoine.
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My work on Python3 and non-ascii paths is done

[M.-A. Lemburg]

Victor Stinner wrote:
> Hi,
> 
> Seven months after my first commit related to this issue, the full test suite 
> of Python 3.2 pass with ASCII, ISO-8859-1 and UTF-8 locale encodings in a non-
> ascii source directory. It means that Python 3.2 now process correctly 
> filenames in all modules, build scripts and other utilities, with any locale 
> encoding.
> 
> 
> General changes:
> 
>  * Encode/decode filenames with the locale encoding, instead of utf-8,
>until the filesystem encoding is set
>  * mbcs encoding (Windows filesystem encoding) is now strict by default,
>whereas it ignores unencodable characters and replace undecodable bytes
>in Python 3.1. Old behaviour can still be used using the right error
>handler: 'ignore' to encode, 'replace' to decode.
>  * tarfile uses utf-8 encoding on Windows (instead of mbcs), and the
>surrogateescape error handler on all OSes
>  * sys.getfilesystemencoding() cannot be None anymore
>  * Don't accept bytearray as filenames anymore
> 
> 
> Changes of the Python API:
> 
>  * Add os.environb: bytes version of os.environ, os.getenvb() function
>and os.supports_bytes_environ constant
>  * Add os.fsencode() and os.fsdecode() functions
>  * Remove sys.setfilesystemencoding() function
> 
> 
> Changes of the C API:
> 
>  * Add PyUnicode_EncodeFSDefault() function
>  * Add PyUnicode_FSDecoder() ParseTuple converter
>  * Add PySys_FormatStdout(), PySys_FormatStderr() and PyErr_WarnFormat()
>functions
>  * Add PyUnicode_AsWideCharString() function: don't need a buffer size.
>  * Add Py_UNICODE_strrchr(), Py_UNICODE_strcat(), PyUnicode_AsUnicodeCopy()
>and Py_UNICODE_strncmp() functions
>  * PyUnicode_DecodeFSDefault() and PyUnicode_DecodeFSDefaultAndSize() use the
>surrogateescape error handler
>  * File utilities: add _Py_wchar2char() (reverse of Py_char2wchar()),
>_Py_stat() and _Py_fopen() functions; move all file utilities to
>Python/fileutils.c
>  * The format string of PyUnicode_FromFormat() and PyErr_Format() is now
>pure ASCII: raise an error on non-ascii character
>  * PyUnicode_FSConverter() doesn't accept bytearray anymore
> 
> 
> Bugfixes:
> 
>  * Fix modules: tarfile, pickle, pickletools, ctypes, subprocess, bz2, ssl,
>profile, xmlrpclib, platform, libpython (gdb plugin), sqlite,
>distutils.log, locale, _warnings, zipimport, imp
>  * Fix functions: os.exec*(), os.system(), ctypes.dlopen(), os.getenv(),
>os.get_exec_path()
>  * Fix tests: test_gdb, test_httpservers, test_cmd_line, test_size,
>test_generic_path, test_subprocess, test_doctest, test_cmd_line_script
>  * Fix utf-8 encoder to support error handlers producing unicode string 
>(eg. 'backslashreplace')
>  * Fix conversion from unicode to a wide character string if Py_UNICODE 
>and wchar_t have different sizes: UTF-16 => UTF-32 or UTF-32 => UTF-16
>  * Fix Python command line parser if the the command line contains surrogates
>  * Avoid _PyUnicode_AsString() because it returns NULL if the string contains
>surrogates, or catch the error
>  * Fix regrtest.py to support surrogate characters in the current working
>directory and in the tracebacks
> 
> 
> I wrote also some tests and documentation.
> 
> The most difficult part was to debug Python initialization (Py_InitializeEx 
> and calculate_path) and the import machinery (import.c, zipimport.c), because 
> gdb does sometimes crash (for various reasons) and because  the import 
> machinery is fragile and difficult to understand.
> 
> A special thanks to Marc-Andre Lemburg, Martin v. Löwis, Antoine Pitrou and 
> Amaury Forgeot d'Arc for their help, useful advices and code reviews!

Many thanks to you for opening up this can of worms and fighting
through all the issues !

> -- Bonus: short story of PYTHONFSENCODING ---
> 
> In the middle of August, I created the PYTHONFSENCODING environment variable, 
> as suggested by Marc-Andre Lemburg. Because of this variable and because 
> Python used utf-8 until the filesystem encoding is known, I had to write ugly 
> and fragile "redecode" functions to redecode all filenames of all objects 
> (sys.path, sys.meta_path, sys.executable, sys.modules, all code objects, 
> etc.).
> 
> Then I found 4 issues related to PYTHONFSENCODING, inconsistencies between 
> the 
> filesystem encoding and the locale encoding. It was not easy to decide how to 
> fix these issues, but at the end, we choosed to drop PYTHONFSENCODING 
> variable, use the locale encoding as the filesystem encoding, and always use 
> utf-8 as the filesystem encoding on Mac OS X.

Time will tell whether we can manage without some logic to tell
Python what to use as file system encoding without having
to rely on the locale settings. Anything is better than having
Python stop with a fatal error, though :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 19 2010)
>>> Python/Zope Consulting and Support ...http://www.egenix.com/
>>> 

Re: [Python-Dev] My work on Python3 and non-ascii paths is done

[Nick Coghlan]

On Tue, Oct 19, 2010 at 11:53 AM, Victor Stinner
 wrote:
> Hi,
>
> Seven months after my first commit related to this issue, the full test suite
> of Python 3.2 pass with ASCII, ISO-8859-1 and UTF-8 locale encodings in a non-
> ascii source directory. It means that Python 3.2 now process correctly
> filenames in all modules, build scripts and other utilities, with any locale
> encoding.

Well done on facing up to that particular dragon and slaying it.
Especially this part:

> the import machinery (import.c, zipimport.c), because
> gdb does sometimes crash (for various reasons) and because  the import
> machinery is fragile and difficult to understand.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My work on Python3 and non-ascii paths is done

[Barry Warsaw]

On Oct 19, 2010, at 03:53 AM, Victor Stinner wrote:

>Seven months after my first commit related to this issue, the full test suite 
>of Python 3.2 pass with ASCII, ISO-8859-1 and UTF-8 locale encodings in a non-
>ascii source directory. It means that Python 3.2 now process correctly 
>filenames in all modules, build scripts and other utilities, with any locale 
>encoding.

Very impressive.  Congratulations, and thanks for following through on what
must have been some tricky work in many of Python's deep dark corners.

Going forward, is there adequate documentation, guidelines, and safeguards for
future coders so that they Do The Right Thing with new code?  Perhaps a short
How To in the standard documentation would be helpful, with links to it from
any old/bad API calls?

>The most difficult part was to debug Python initialization (Py_InitializeEx 
>and calculate_path) and the import machinery (import.c, zipimport.c), because 
>gdb does sometimes crash (for various reasons) and because  the import 
>machinery is fragile and difficult to understand.

You are also a master of the understatement! :)

-Barry


signature.asc
Description: PGP signature
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Exposing pkguitl's import emulation (was Re: [Python-checkins] r85538 - python/branches/py3k/Doc/library/pkgutil.rst)

[P.J. Eby]

At 08:03 AM 10/18/2010 +1000, Nick Coghlan wrote:

I'm a little dubious about exposing these officially. They're mainly a
hack to get some parts of the standard library working (e.g. runpy) in
the absence of full PEP 302 support in the imp module, not really
something we want to encourage anyone else to use (and yes, they
should probably have underscores in their names, but we missed that
when the various private implementations scattered around the stdlib
were consolidated in pkgutil).


Well, my intention at least was that they should be documented and 
released; it's the documenting part I didn't get around to.  ;-)


Of course, this was also pre-importlib; were we starting the work 
today, the obvious thing to do would be to expose the Python 
implementations of the relevant objects.




That said, who knows when we'll actually have it done right, so in the
meantime maybe having an official workaround is better than nothing...

Cheers,
Nick.

--
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] Support for async read/write

[Jesus Cea]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Current Python lacks support for "aio_*" syscalls to do async IO. I
think this could be a nice addition for python 3.3.

If you agree, I will create an issue in the tracker. If you think the
idea is of no value, please say so for me to move on. Maybe an 3th party
module, but I think this functionality sould be available in core python.

Thanks!.

PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read,
aio_return, aio_write.

- -- 
Jesus Cea Avion _/_/  _/_/_/_/_/_/
j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
jabber / xmpp:j...@jabber.org _/_/_/_/  _/_/_/_/_/
.  _/_/  _/_/_/_/  _/_/  _/_/
"Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBTL3MUplgi5GaxT1NAQKv5QQAnDan88kXJ67fucz2rT/mZze+065lm9E4
+XJ2JfqGMVE1/qMsXwg81l19RHSYReBgBjd7zyXWE9Fk/1Rfq4gjOZEtoG0IpGZG
E3CH+Ab5/o/PjJZNKQaPpe0cwGSXFPKkPpgepKS1d8ZXyf6VXMc8UTSWjMI5jIVe
4M+yvw5m4I0=
=nsdO
-END PGP SIGNATURE-
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Peter Portante]

Yes, that would be a nice addition. -peter


On 10/19/10 12:50 PM, "Jesus Cea"  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Current Python lacks support for "aio_*" syscalls to do async IO. I
> think this could be a nice addition for python 3.3.
> 
> If you agree, I will create an issue in the tracker. If you think the
> idea is of no value, please say so for me to move on. Maybe an 3th party
> module, but I think this functionality sould be available in core python.
> 
> Thanks!.
> 
> PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read,
> aio_return, aio_write.
> 
> - -- 
> Jesus Cea Avion _/_/  _/_/_/_/_/_/
> j...@jcea.es - http://www.jcea.es/ _/_/_/_/  _/_/_/_/  _/_/
> jabber / xmpp:j...@jabber.org _/_/_/_/  _/_/_/_/_/
> .  _/_/  _/_/_/_/  _/_/  _/_/
> "Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
> 
> iQCVAwUBTL3MUplgi5GaxT1NAQKv5QQAnDan88kXJ67fucz2rT/mZze+065lm9E4
> +XJ2JfqGMVE1/qMsXwg81l19RHSYReBgBjd7zyXWE9Fk/1Rfq4gjOZEtoG0IpGZG
> E3CH+Ab5/o/PjJZNKQaPpe0cwGSXFPKkPpgepKS1d8ZXyf6VXMc8UTSWjMI5jIVe
> 4M+yvw5m4I0=
> =nsdO
> -END PGP SIGNATURE-
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/peter.a.portante%40gmail.com


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[exarkun]

On 04:50 pm, j...@jcea.es wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Current Python lacks support for "aio_*" syscalls to do async IO. I
think this could be a nice addition for python 3.3.


Adding more platform wrappers is always nice.  Keep in mind that the 
quality of most (all?) aio_* implementations is spotty at best, though. 
On Linux, they will sometimes block (for example, if you fail to align 
buffers properly, or open a file without O_DIRECT, or if there are too 
many other aio operations active on the system at the time, etc).  Thus, 
these APIs are finicky at best, and the Python bindings will be 
similarly fragile.


Jean-Paul

If you agree, I will create an issue in the tracker. If you think the
idea is of no value, please say so for me to move on.


Do you have an application in mind?

Maybe an 3th party
module, but I think this functionality sould be available in core 
python.


Starting off as a 3rd party library to try to develop some interest and 
users (and thus experience) before adding it to the standard library 
might make sense (as it frequently does).

Thanks!.

PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read,
aio_return, aio_write.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Giampaolo Rodolà]

You should  file a new issue on the bug tracker but unless you have a
patch to propose it's unlikely that someone else is gonna implement
it.


Regards

--- Giampaolo
http://code.google.com/p/pyftpdlib/
http://code.google.com/p/psutil/


2010/10/19 Jesus Cea :
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Current Python lacks support for "aio_*" syscalls to do async IO. I
> think this could be a nice addition for python 3.3.
>
> If you agree, I will create an issue in the tracker. If you think the
> idea is of no value, please say so for me to move on. Maybe an 3th party
> module, but I think this functionality sould be available in core python.
>
> Thanks!.
>
> PS: The function calls are: aio_cancel, aio_error, aio_fsync, aio_read,
> aio_return, aio_write.
>
> - --
> Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
> j...@jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
> jabber / xmpp:j...@jabber.org         _/_/    _/_/          _/_/_/_/_/
> .                              _/_/  _/_/    _/_/          _/_/  _/_/
> "Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQCVAwUBTL3MUplgi5GaxT1NAQKv5QQAnDan88kXJ67fucz2rT/mZze+065lm9E4
> +XJ2JfqGMVE1/qMsXwg81l19RHSYReBgBjd7zyXWE9Fk/1Rfq4gjOZEtoG0IpGZG
> E3CH+Ab5/o/PjJZNKQaPpe0cwGSXFPKkPpgepKS1d8ZXyf6VXMc8UTSWjMI5jIVe
> 4M+yvw5m4I0=
> =nsdO
> -END PGP SIGNATURE-
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com
>
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My work on Python3 and non-ascii paths is done

[Georg Brandl]

Am 19.10.2010 16:12, schrieb Barry Warsaw:
> On Oct 19, 2010, at 03:53 AM, Victor Stinner wrote:
> 
>>Seven months after my first commit related to this issue, the full test suite 
>>of Python 3.2 pass with ASCII, ISO-8859-1 and UTF-8 locale encodings in a non-
>>ascii source directory. It means that Python 3.2 now process correctly 
>>filenames in all modules, build scripts and other utilities, with any locale 
>>encoding.
> 
> Very impressive.  Congratulations, and thanks for following through on what
> must have been some tricky work in many of Python's deep dark corners.

I'd like to join that statement.  It must be like coming back to the surface
after a long trip to deep underground caves with lava and sticky air :)

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[James Y Knight]

On Oct 19, 2010, at 1:47 PM, exar...@twistedmatrix.com wrote:

> Adding more platform wrappers is always nice.  Keep in mind that the quality 
> of most (all?) aio_* implementations is spotty at best, though. On Linux, 
> they will sometimes block (for example, if you fail to align buffers 
> properly, or open a file without O_DIRECT, or if there are too many other aio 
> operations active on the system at the time, etc).  

You're thinking of the linux-specific AIO calls. Those have the properties 
you're describing (which makes them pretty useless for most code too), but 
they're completely different from the aio_* functions.

The POSIX aio_* calls don't do any of that. They aren't syscalls implemented in 
the kernel, they're implemented in glibc. They "simply" create a threadpool in 
your process to call the standard synchronous operations, and make it difficult 
to reliably get completion notification (completion notification takes place 
via Real-Time signals (SIGEV_SIGNAL), which can be dropped if linux runs out of 
space in its RT-signal-queue, and when that happens you get no indication that 
that has occurred. You can also do completion notification via calling a 
function on a thread (SIGEV_THREAD), but, for that, glibc will always spawns a 
brand new thread for each notification, which is quite slow.)

Basically: you shouldn't ever use those APIs. Especially on linux, but probably 
everywhere else. 

So, in conclusion, I disagree that adding wrappers for these would be nice. It 
wouldn't. It would cause some people to think they would be useful things to 
call, and they would always be wrong.

James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] r85724 - in python/branches/py3k: Lib/logging/__init__.py Misc/NEWS

[Georg Brandl]

Am 19.10.2010 17:26, schrieb vinay.sajip:
> Author: vinay.sajip
> Date: Tue Oct 19 17:26:24 2010
> New Revision: 85724
> 
> Log:
> logging: Added _logRecordClass, getLogRecordClass, setLogRecordClass to 
> increase flexibility of LogRecord creation.

Will this be documented?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Antoine Pitrou]

Hello Jesus,

> Current Python lacks support for "aio_*" syscalls to do async IO. I
> think this could be a nice addition for python 3.3.
> 
> If you agree, I will create an issue in the tracker. If you think the
> idea is of no value, please say so for me to move on. Maybe an 3th party
> module, but I think this functionality sould be available in core python.

Well, if you think this is interesting, it would be nice if you could
do a bit more research instead of just filing issues for us to solve.
After all, you probably already have an idea about why this is useful,
how to interface it into the stdlib, etc. Just saying "we should expose
these functions" doesn't bring a lot of added value IMO (there are many
system functions which could be exposed and currently aren't).

Also, the canonical way to do file I/O in Python 3 is the `io` lib,
therefore it would be a bit of a shame to have separate, non-integrated
`aio_*` functions.

Also, since the `io` lib is already supposed to support non-blocking
IO, perhaps it would be valuable to stress this support and propose any
interesting patches for fixing and/or improving it.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Martin v. Löwis]

Am 19.10.2010 19:47, schrieb exar...@twistedmatrix.com:
> On 04:50 pm, j...@jcea.es wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Current Python lacks support for "aio_*" syscalls to do async IO. I
>> think this could be a nice addition for python 3.3.
> 
> Adding more platform wrappers is always nice. 

Exactly so. Exposing platform APIs as-is really doesn't need much
discussion, except when the platform wrapper is going to deviate
significantly from the platform API. There is a long tradition of
exposing even obscure unixish APIs in Python, making it the #1 language
to write exotic scripts in. We definitely want to continue this
tradition.

Please refrain from trying to come up with "abstractions", though;
this should be outside of Python for the moment. Covering both Unix
AIO and Windows completion ports (say) may be quite a challenge
(but then, JP may have some view on this).

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Martin v. Löwis]

> So, in conclusion, I disagree that adding wrappers for these would be
> nice. It wouldn't. It would cause some people to think they would be
> useful things to call, and they would always be wrong.

We are all consenting adults. If people want to shoot themselves in
their feet, we let them. For example, we have os.open, even though
there is no garbage collection for file handles, and we have
os._exit, even though it doesn't call finalizers.

People should trust that the APIs we expose go *literally* into the
library, so we can blame any consequences that this has onto the
library (and possibly the kernel, in turn).
People readily accept that explanation.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Martin v. Löwis]

> Also, the canonical way to do file I/O in Python 3 is the `io` lib,
> therefore it would be a bit of a shame to have separate, non-integrated
> `aio_*` functions.

I disagree. We also have posix.open, posix.dup, etc. We always expose
POSIX functions in the posix module (except for the socket functions,
unfortunately, and a few other exceptions), and I see no reason to break
with this tradition. The io module should be thought of as sitting
on top of the posix module (and it initially was).

> Also, since the `io` lib is already supposed to support non-blocking
> IO, perhaps it would be valuable to stress this support and propose any
> interesting patches for fixing and/or improving it.

I think that's entirely independent.

Regards,
Martin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Antoine Pitrou]

Le mercredi 20 octobre 2010 à 00:48 +0200, "Martin v. Löwis" a écrit :
> > Also, the canonical way to do file I/O in Python 3 is the `io` lib,
> > therefore it would be a bit of a shame to have separate, non-integrated
> > `aio_*` functions.
> 
> I disagree. We also have posix.open, posix.dup, etc. We always expose
> POSIX functions in the posix module (except for the socket functions,
> unfortunately, and a few other exceptions), and I see no reason to break
> with this tradition. The io module should be thought of as sitting
> on top of the posix module (and it initially was).

I'm not suggesting to break with any "tradition", but that building
duplicate APIs to do file I/O has little value and only makes things
more cumbersome.

> > Also, since the `io` lib is already supposed to support non-blocking
> > IO, perhaps it would be valuable to stress this support and propose any
> > interesting patches for fixing and/or improving it.
> 
> I think that's entirely independent.

I don't think it is. If the goal is to give ways to improve file I/O
performance for specialized use cases, then it certainly deserves
integration with the current I/O stack. If the goal is not that, then I
don't really know what it is.

It would be nice to know about the use case Jesus has in mind.

Regards

Antoine.


___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[James Y Knight]

On Oct 19, 2010, at 6:44 PM, Martin v. Löwis wrote:

>> So, in conclusion, I disagree that adding wrappers for these would be
>> nice. It wouldn't. It would cause some people to think they would be
>> useful things to call, and they would always be wrong.
> 
> We are all consenting adults. If people want to shoot themselves in
> their feet, we let them. For example, we have os.open, even though
> there is no garbage collection for file handles, and we have
> os._exit, even though it doesn't call finalizers.


There's a difference.

os._exit is useful. os.open is useful. aio_* are *not* useful. For anything. If 
there's anything you think you want to use them for, you're wrong. It either 
won't work properly or it will worse performing than the simpler alternatives.

It would absolutely be a waste of time (of both the implementor of the wrapper 
and the poor users who stumble across them in documentation and try to use 
them) to bother adding wrappers to these functions for python. 

James
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] My work on Python3 and non-ascii paths is done

[Victor Stinner]

Le mardi 19 octobre 2010 16:12:56, Barry Warsaw a écrit :
> Going forward, is there adequate documentation, guidelines, and safeguards
> for future coders so that they Do The Right Thing with new code?  Perhaps
> a short How To in the standard documentation would be helpful, with links
> to it from any old/bad API calls?

Hum, as usual, I suggest to decode all inputs to unicode as early as possible, 
and encode back to bytes (or other native format) at the last moment. For 
filenames, it means that PyUnicode_FSDecoder() is better than 
PyUnicode_FSConverter(), because it gives an unicode object (instead of byte 
string) and so the function will support unencodable characters.

Use PyUnicode_EncodeFSDefault() / PyUnicode_DecodeFSDefault() and 
os.fsencode() / os.fsdecode() to encode/decode filenames instead of your own 
function, to support the PEP 383 (undecodable bytes <=> surrogate characters).

Be also careful to support undecodable bytes (on OSes other than Windows), eg. 
try a filename with a non-ASCII character with the C locale (ASCII locale 
encoding). Even with utf-8 filesystem encoding, this problem may occurs with a 
system not correclty configured (eg. USB key with the FAT fileystem using the 
"wrong" encoding).

If you would like to avoid all encoding issues on filenames on UNIX/BSD, use 
bytes: os.environb, os.listdir(b'.'), os.getcwdb(), etc.

Be careful with the utf-8 codec: its default mode (strict error handler) 
refuses to encode surrogate characters. Eg. print(filename) may raise a 
UnicodeEncodeError. Use repr(filename) to escape surrogate characters.

--

I plan to fix Python documentation: specify the encoding used to decode all 
byte string arguments of the C API. I already wrote a draft patch: issue 
#9738. This lack of documentation was a big problem for me, because I had to 
follow the function calls to get the encoding.

-- 
Victor Stinner
http://www.haypocalc.com/
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] [patch] fpconst for python3

[Neal Becker]

Where should I send this patch?

diff -u fpconst-0.7.2/fpconst.py fpconst-0.7.2.new/fpconst.py
--- fpconst-0.7.2/fpconst.py2005-02-24 12:42:03.0 -0500
+++ fpconst-0.7.2.new/fpconst.py2010-10-19 20:55:07.407765664 -0400
@@ -40,18 +40,18 @@
 ident = "$Id: fpconst.py,v 1.16 2005/02/24 17:42:03 warnes Exp $"
 
 import struct, operator
+from functools import reduce
 
 # check endianess
-_big_endian = struct.pack('i',1)[0] != '\x01'
-
+_big_endian = struct.pack('i',1)[0] != 1
 # and define appropriate constants
 if(_big_endian): 
-NaN= struct.unpack('d', '\x7F\xF8\x00\x00\x00\x00\x00\x00')[0]
-PosInf = struct.unpack('d', '\x7F\xF0\x00\x00\x00\x00\x00\x00')[0]
+NaN= struct.unpack('d', b'\x7F\xF8\x00\x00\x00\x00\x00\x00')[0]
+PosInf = struct.unpack('d', b'\x7F\xF0\x00\x00\x00\x00\x00\x00')[0]
 NegInf = -PosInf
 else:
-NaN= struct.unpack('d', '\x00\x00\x00\x00\x00\x00\xf8\xff')[0]
-PosInf = struct.unpack('d', '\x00\x00\x00\x00\x00\x00\xf0\x7f')[0]
+NaN= struct.unpack('d', b'\x00\x00\x00\x00\x00\x00\xf8\xff')[0]
+PosInf = struct.unpack('d', b'\x00\x00\x00\x00\x00\x00\xf0\x7f')[0]
 NegInf = -PosInf
 
 def _double_as_bytes(dval):

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] [patch] fpconst for python3

[Benjamin Peterson]

fpconst developers?

2010/10/19 Neal Becker :
> Where should I send this patch?
>
> diff -u fpconst-0.7.2/fpconst.py fpconst-0.7.2.new/fpconst.py
> --- fpconst-0.7.2/fpconst.py    2005-02-24 12:42:03.0 -0500
> +++ fpconst-0.7.2.new/fpconst.py        2010-10-19 20:55:07.407765664 -0400
> @@ -40,18 +40,18 @@
>  ident = "$Id: fpconst.py,v 1.16 2005/02/24 17:42:03 warnes Exp $"
>
>  import struct, operator
> +from functools import reduce
>
>  # check endianess
> -_big_endian = struct.pack('i',1)[0] != '\x01'
> -
> +_big_endian = struct.pack('i',1)[0] != 1
>  # and define appropriate constants
>  if(_big_endian):
> -    NaN    = struct.unpack('d', '\x7F\xF8\x00\x00\x00\x00\x00\x00')[0]
> -    PosInf = struct.unpack('d', '\x7F\xF0\x00\x00\x00\x00\x00\x00')[0]
> +    NaN    = struct.unpack('d', b'\x7F\xF8\x00\x00\x00\x00\x00\x00')[0]
> +    PosInf = struct.unpack('d', b'\x7F\xF0\x00\x00\x00\x00\x00\x00')[0]
>     NegInf = -PosInf
>  else:
> -    NaN    = struct.unpack('d', '\x00\x00\x00\x00\x00\x00\xf8\xff')[0]
> -    PosInf = struct.unpack('d', '\x00\x00\x00\x00\x00\x00\xf0\x7f')[0]
> +    NaN    = struct.unpack('d', b'\x00\x00\x00\x00\x00\x00\xf8\xff')[0]
> +    PosInf = struct.unpack('d', b'\x00\x00\x00\x00\x00\x00\xf0\x7f')[0]
>     NegInf = -PosInf
>
>  def _double_as_bytes(dval):
>
> ___
> Python-Dev mailing list
> Python-Dev@python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> http://mail.python.org/mailman/options/python-dev/benjamin%40python.org
>



-- 
Regards,
Benjamin
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Glyph Lefkowitz]

On Oct 19, 2010, at 8:09 PM, James Y Knight wrote:

> There's a difference.
> 
> os._exit is useful. os.open is useful. aio_* are *not* useful. For anything. 
> If there's anything you think you want to use them for, you're wrong. It 
> either won't work properly or it will worse performing than the simpler 
> alternatives.


I'd like to echo this sentiment.  This is not about providing a 'safe' wrapper 
to hide some powerful feature of these APIs: the POSIX aio_* functions are 
really completely useless.

To quote the relevant standard 
:

APPLICATION USAGE

None.

RATIONALE

None.

FUTURE DIRECTIONS

None.

Not only is the performance usually worse than expected, the behavior of aio_* 
functions require all kinds of subtle and mysterious coordination with signal 
handling, which I'm not entirely sure Python would even be able to pull off 
without some modifications to the signal module.  (And, as Jean-Paul mentioned, 
if your OS kernel runs out of space in a queue somewhere, completion 
notifications might just never be delivered at all.)

I would love for someone to prove me wrong.  In particular, I would really love 
for there to be a solution to asynchronous filesystem I/O better than "start a 
thread, read until you block".  But, as far as I know, there isn't, and 
wrapping these functions will just confuse and upset anyone who attempts to use 
them in any way.

___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[exarkun]

On 01:37 am, gl...@twistedmatrix.com wrote:


On Oct 19, 2010, at 8:09 PM, James Y Knight wrote:

There's a difference.

os._exit is useful. os.open is useful. aio_* are *not* useful. For 
anything. If there's anything you think you want to use them for, 
you're wrong. It either won't work properly or it will worse 
performing than the simpler alternatives.



I'd like to echo this sentiment.  This is not about providing a 'safe' 
wrapper to hide some powerful feature of these APIs: the POSIX aio_* 
functions are really completely useless.


To quote the relevant standard 
:



APPLICATION USAGE

None.

RATIONALE

None.

FUTURE DIRECTIONS

None.

Not only is the performance usually worse than expected, the behavior 
of aio_* functions require all kinds of subtle and mysterious 
coordination with signal handling, which I'm not entirely sure Python 
would even be able to pull off without some modifications to the signal 
module.  (And, as Jean-Paul mentioned, if your OS kernel runs out of 
space in a queue somewhere, completion notifications might just never 
be delivered at all.)


Just to be clear, James corrected me there.  I thought Jesus was talking 
about the mostly useless Linux AIO APIs, which have the problems I 
described.  He was actually talking about the POSIX AIO APIs, which have 
a different set of problems making them a waste of time.


Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] Support for async read/write

[Jeffrey Yasskin]

On Tue, Oct 19, 2010 at 8:37 PM, Glyph Lefkowitz
 wrote:
> I'd like to echo this sentiment.  This is not about providing a 'safe'
> wrapper to hide some powerful feature of these APIs: the POSIX aio_*
> functions are really completely useless.
> To quote the relevant standard
> :
>
> APPLICATION USAGE
> None.
> RATIONALE
> None.
> FUTURE DIRECTIONS
> None.


No comment on the rest of your claim, but this is a silly argument.
The standard says the same thing about at least fcntl.h, signal.h,
pthread.h, and ucontext.h, which clearly are useful.

Jeffrey
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com