Is this the right way to write a codec error handler?
I want an error handler that falls back on Latin-1 for anything which
cannot be decoded. Is this the right way to write it?
def latin1_fallback(exception):
assert isinstance(exception, UnicodeError)
start, end = exception.start, exception.end
obj = exception.object
if isinstance(exception, UnicodeDecodeError):
return (obj[start:end].decode('latin1'), end+1)
elif isinstance(exception, UnicodeEncodeError):
return (obj[start:end].encode('latin1'), end+1)
else:
raise
Comments, corrections and criticisms welcome.
Thanks.
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: Is this the right way to write a codec error handler?
20.01.18 10:32, Steven D'Aprano пише:
I want an error handler that falls back on Latin-1 for anything which
cannot be decoded. Is this the right way to write it?
def latin1_fallback(exception):
assert isinstance(exception, UnicodeError)
start, end = exception.start, exception.end
obj = exception.object
if isinstance(exception, UnicodeDecodeError):
return (obj[start:end].decode('latin1'), end+1)
elif isinstance(exception, UnicodeEncodeError):
return (obj[start:end].encode('latin1'), end+1)
else:
raise
Just `end` instead of `end+1`.
And it is safer to use `bytes.decode(obj[start:end], 'latin1')` or
`str(obj[start:end], 'latin1')` instead of
`obj[start:end].decode('latin1')`. Just for the case if obj has
overridden decode() method.
Otherwise LGTM.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fourth example from PEP 342
On 19/01/18 22:50, Léo El Amri wrote:
> Hello list,
>
> I am currently trying to learn co-routine/asynchronous mechanisms in
> Python. I read the PEP 342, but I stumble on the fourth example.
> I don't understand what the lines "data = yield nonblocking_read(sock)"
> in echo_handler() and "connected_socket = yield
> nonblocking_accept(sock)" in listen_on() are trying to do.
>
> For example, can someone explain me how the returned value in the line
> "connected_socket = yield nonblocking_accept(sock)" can be used on the
> next line ("trampoline.add(handler(connected_socket))") ? To me, it
> looks like the returned value is lost in the Trampoline, when resume()
> gets the returned value of the yield expression.
>
> Léo
>
Let's see.
Upon the line
connected_socket = yield nonblocking_accept(sock)
control is returned to t.resume. nonblocking_accept is supposed to be a
coroutine, so t.resume does this:
if isinstance(value, types.GeneratorType):
# Yielded to a specific coroutine, push the
# current one on the stack, and call the new
# one with no args
self.schedule(value, (coroutine,stack))
Ergo, it schedules the nonblocking_accept coroutine (‘value’) to be
called on the next iteration, and keeps the running listen_on coroutine
(‘coroutine’) on the stack for safekeeping.
On a subsequent iteration (jump?) of the trampoline, nonblocking_accept
will yield a socket. Then, resume will
elif stack:
# Yielded a result, pop the stack and send the
# value to the caller
self.schedule(stack[0], stack[1], value)
plan to return control to listen_on (‘stack[0]’), and (the next time
around) send the value back in.
if exc:
value = coroutine.throw(value,*exc)
else:
value = coroutine.send(value)
Now, listen on is running again and has the value yielded by
nonblocking_accept.
I wish I knew how exactly nonblocking_accept was supposed to work here,
but that's beside the point as the world of Python coroutines and async
I/O has moved on since 2005.
Hope this helps
Thomas
--
https://mail.python.org/mailman/listinfo/python-list
Re: Fourth example from PEP 342
On 20/01/2018 11:55, Thomas Jollans wrote: > control is returned to t.resume. nonblocking_accept is supposed to be a > coroutine > Ergo, it schedules the nonblocking_accept coroutine (‘value’) to be > called on the next iteration, and keeps the running listen_on coroutine > (‘coroutine’) on the stack for safekeeping. Hello Thomas, I missed THE point of the examples. nonblocking_accept is a coroutine. I was stuck in the C way of doing non blocking, and I was wondering what was the magic behind the examples. Perfect explanation, now I understand. Still, I also wonder how theses "nonblocking_" functions are meant to be implemented. But it's another topic. Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: How to use asyncore with SSL?
On 2018-01-18, Grant Edwards wrote: [regarding secure-smtpd -- a module based on smtpd and asyncore] > That makes the SSL support pretty much useless. > > I'm trying to fix that, but I can't find any information or > documentation about using asyncore with SSL. Asyncore seems to be based on fundamental assumptions that aren't true for non-blocking ssl sockets. After looking into it for a couple hours, the way you use ssl with asyncore is like this: pid = subprocess.Pipe(["stunnel","stunnel.conf"]).pid # insert asyncore app here os.kill(pid,signal.SIGTERM) -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Why is there no functional xml?
Looking around for how to create (l)xml one sees typical tutorials like this:
https://www.blog.pythonlibrary.org/2013/04/30/python-101-intro-to-xml-parsing-with-elementtree/
Given the requirement to build up this xml:
1181251680
04008200E000
1181572063
1800
Bring pizza home
the way I would rather do it is thus:
[Note in actual practice the 'contents' such as 1181251680 etc would come
from suitable program variables/function-calls
]
ex = Ea("zAppointments", {'reminder':'15'},
E("appointment",
En("begin", 1181251680),
Et("uid", "04008200E000"),
En("alarmTime", 1181572063),
E("state"),
E("location"),
En("duration",1800),
Et("subject", "Bring pizza home")))
with the following obvious definitions:
[The function names are short so that the above becomes correspondingly
readable]
from lxml.etree import Element
def Ea(tag, attrib=None, *subnodes):
"xml node constructor"
root = Element(tag, attrib)
for n in subnodes:
root.append(n)
return root
def E(tag, *subnodes):
"Like E but without attributes"
root = Element(tag)
for n in subnodes:
root.append(n)
return root
def Et(tag, text):
"A pure text node"
root = E(tag)
root.text = text
return root
def En(tag, text):
"A node containing a integer"
root = E(tag)
root.text = str(text)
return root
This approach seems so obvious that I find it hard to believe its not there
somewhere…
Am I missing something??
--
https://mail.python.org/mailman/listinfo/python-list
Re: Speeding up the implementation of Stochastic Gradient Ascent in Python
Aside from the obvious imports I've added (import numpy as np, etc),
there are still undefined objects (`names`, `times_`, `feat`,
`bind_ind`) and indentation errors. Can you post a *working* code to be
sped up and a benchmarking test?
As you're using Python 2, you can improve the code with the following:
- Use `xrange` in place of `range`.
- Printing in loops dramatically slows things down.
- Loop directly over an iterable (and maybe use
`collections.defaultdict`):
# This:
def pos_per_user_(self):
pos_per_user = {}
for k, v in self.userItems.items():
for i in range(len(self.userItems[k])):
item__ = self.userItems[k][i][0]
if k not in pos_per_user:
pos_per_user[k] = [item__]
else:
pos_per_user[k].append(item__)
return pos_per_user
# Becomes like this:
from collections import defaultdict
def pos_per_user(self):
pos_per_user = defaultdict(list)
for k, v in self.userItems.iteritems():
for item in items:
pos_per_user[k].append(item[0])
return pos_per_user
- You also have too many list creations inside loops, which is also slow:
# If you do nothing, use xrange here for it will not construct
# a list. Avoid materializing a list if all you do is iterate.
for bin in range(10):
for i in range(len(self.list_of_items)):
for k in range(self.K2):
for f in range(4096):
--
~ Jugurtha Hadjar,
--
https://mail.python.org/mailman/listinfo/python-list
Re: How to use asyncore with SSL?
Grant Edwards : > Asyncore seems to be based on fundamental assumptions that aren't true > for non-blocking ssl sockets. Pot calling kettle black. OpenSSL isn't the easiest beast to deal with, but I have been able to abstract it (in C) so it behaves very close to TCP. The one blemish is in the fact that the TLS protocol does not support a half-duplex connection. Shame. The WANT_READ/WANT_WRITE silliness should be abstracted out of the non-blocking TLS library so the application doesn't need to know anything about it. Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: How to use asyncore with SSL?
On 2018-01-20, Marko Rauhamaa wrote: > Grant Edwards : > >> Asyncore seems to be based on fundamental assumptions that aren't true >> for non-blocking ssl sockets. > > Pot calling kettle black. > > OpenSSL isn't the easiest beast to deal with, but I have been able to > abstract it (in C) so it behaves very close to TCP. The one blemish is > in the fact that the TLS protocol does not support a half-duplex > connection. Shame. > > The WANT_READ/WANT_WRITE silliness should be abstracted out of the > non-blocking TLS library so the application doesn't need to know > anything about it. I won't argue with that. I think that non-blocking ssl-wrapped sockets _should_ have the same select/poll/send/recv API/semantics that normal sockets do. I thought about writing my own wrapped-ssl-socket class that does that, but using stunnel was just so much easier. If you _did_ want to wrap sockets like that, I think you'd need to actually run a thread to deal with the SSL socket and provide a "proxy" socket or pipe for use with select/poll. Basically you'd be doing what stunnel does only doing it in-process. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: How to use asyncore with SSL?
Grant Edwards : > On 2018-01-20, Marko Rauhamaa wrote: >> OpenSSL isn't the easiest beast to deal with, but I have been able to >> abstract it (in C) so it behaves very close to TCP. The one blemish >> is in the fact that the TLS protocol does not support a half-duplex >> connection. Shame. >> >> The WANT_READ/WANT_WRITE silliness should be abstracted out of the >> non-blocking TLS library so the application doesn't need to know >> anything about it. > > I won't argue with that. I think that non-blocking ssl-wrapped > sockets _should_ have the same select/poll/send/recv API/semantics > that normal sockets do. I thought about writing my own > wrapped-ssl-socket class that does that, but using stunnel was just so > much easier. If you _did_ want to wrap sockets like that, I think > you'd need to actually run a thread to deal with the SSL socket and > provide a "proxy" socket or pipe for use with select/poll. > > Basically you'd be doing what stunnel does only doing it in-process. Stunnel is fine for many applications but not for our needs. Also, a subsidiary thread is not necessary. Everything can be done within an async framework (in C anyway). Marko -- https://mail.python.org/mailman/listinfo/python-list
Can't get python running
I downloaded python 3.6.4 and although everything about the installation seems correct (path, file size, checking on cmd to see if file installed correctly-it is) the only window that will pop up when I attempt to run it is the set-up window offering to modify, restore, or uninstall. I have uninstalled and re-installed the program several times, run the repair function many times and nothing changes. Would someone please help me or direct me to someone who can? Thank you. -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
On 1/20/2018 12:16 PM, Jim Sadler wrote: I downloaded python 3.6.4 and although everything about the installation seems correct (path, file size, checking on cmd to see if file installed correctly-it is) the only window that will pop up when I attempt to run it is the set-up window offering to modify, restore, or uninstall. The only time that should pop up is when you rerun the installer instead of the installed binary. (Or when you invoke it from the installed programs control panel.) *Exactly* how are you trying to run the installed Python. What version of Windows? -- Terry Jan Reedy -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
Hi, Jim, On Sat, Jan 20, 2018 at 11:16 AM, Jim Sadler wrote: > I downloaded python 3.6.4 and although everything about the installation > seems correct (path, file size, checking on cmd to see if file installed > correctly-it is) the only window that will pop up when I attempt to run it > is the set-up window offering to modify, restore, or uninstall. I have > uninstalled and re-installed the program several times, run the repair > function many times and nothing changes. Would someone please help me or > direct me to someone who can? How exactly are attempting to run python? Thank you. > Thank you. > -- > https://mail.python.org/mailman/listinfo/python-list -- https://mail.python.org/mailman/listinfo/python-list
Re: How to use asyncore with SSL?
On 2018-01-20, Marko Rauhamaa wrote: > Grant Edwards : [...] >> I won't argue with that. I think that non-blocking ssl-wrapped >> sockets _should_ have the same select/poll/send/recv API/semantics >> that normal sockets do. I thought about writing my own >> wrapped-ssl-socket class that does that, but using stunnel was just so >> much easier. If you _did_ want to wrap sockets like that, I think >> you'd need to actually run a thread to deal with the SSL socket and >> provide a "proxy" socket or pipe for use with select/poll. >> >> Basically you'd be doing what stunnel does only doing it in-process. > > Stunnel is fine for many applications but not for our needs. Also, a > subsidiary thread is not necessary. Everything can be done within an > async framework (in C anyway). Of course it can. But (ISTM) either the async framework has to be SSL-aware, or the sockets have to be wrapped and "proxied" using another socket or pipe -- I don't see how else you can provide something that has the same select/poll and send/recv semantacs that an unwrapped, non-blocking socket provides. -- Grant -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
On 20/01/2018 17:16, Jim Sadler wrote: I downloaded python 3.6.4 and although everything about the installation seems correct (path, file size, checking on cmd to see if file installed correctly-it is) What do you mean by 'checking on cmd'? I would install it somewhere like c:\python364, then it is easier to find than having it buried in its default installation path (I assume this is Windows). If it's installed properly, then python.exe should be present in c:\python364, and you can start it with: c:\python364\python or: c: cd \python364 python I assume you can open a command prompt. If not in python364, then substitute the actual installation path. Or click Start, and type python.exe into the search box. If it finds it, click it. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
On Sun, Jan 21, 2018 at 12:08 PM, bartc wrote: > On 20/01/2018 17:16, Jim Sadler wrote: >> >> I downloaded python 3.6.4 and although everything about the installation >> seems correct (path, file size, checking on cmd to see if file installed >> correctly-it is) > > > What do you mean by 'checking on cmd'? > > I would install it somewhere like c:\python364, then it is easier to find > than having it buried in its default installation path (I assume this is > Windows). python36 is a better choice of name, as it doesn't cause problems when you upgrade to a new bugfix release. But there are permissions issues with dropping stuff straight into the root directory, which is why the default installers now put Python into Program Files. Jim, let the installer put it where it wants to, and make sure you've added it to PATH. Then you should be able to type "py" to start Python. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
On 21/01/2018 01:21, Chris Angelico wrote: On Sun, Jan 21, 2018 at 12:08 PM, bartc wrote: On 20/01/2018 17:16, Jim Sadler wrote: I downloaded python 3.6.4 and although everything about the installation seems correct (path, file size, checking on cmd to see if file installed correctly-it is) What do you mean by 'checking on cmd'? I would install it somewhere like c:\python364, then it is easier to find than having it buried in its default installation path (I assume this is Windows). python36 is a better choice of name That's what I use; I assumed everyone else used the 3-digit version in the path. , as it doesn't cause problems when you upgrade to a new bugfix release. But there are permissions issues with dropping stuff straight into the root directory, which is why the default installers now put Python into Program Files. It's not in the root, the executable will be: c:\python36\python.exe Not c:\python.exe. Yes, Windows allows you to have your own directories within /. It's not Unix. Jim, let the installer put it where it wants to, and make sure you've added it to PATH. Then you should be able to type "py" to start Python. If I try to install 3.7 (as I already have 3.6) it suggests putting it in: c:\Users\users\AppData\Local\Programs\Python\Python37-32 Not actually a snappy path if you need to get there in a hurry to sort out problems. That is, in a location 7 levels deep. Get it working anywhere first to find out what the problem is. -- bartc -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
On Sun, 21 Jan 2018 12:21:40 +1100, Chris Angelico wrote: > Jim, let the installer put it where it wants to, and make sure you've > added it to PATH. Then you should be able to type "py" to start Python. Shouldn't the installer ensure that it puts "py" somewhere on the path? -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
On Sun, Jan 21, 2018 at 12:40 PM, bartc wrote: > On 21/01/2018 01:21, Chris Angelico wrote: >> >> On Sun, Jan 21, 2018 at 12:08 PM, bartc wrote: >>> >>> On 20/01/2018 17:16, Jim Sadler wrote: I downloaded python 3.6.4 and although everything about the installation seems correct (path, file size, checking on cmd to see if file installed correctly-it is) >>> >>> >>> >>> What do you mean by 'checking on cmd'? >>> >>> I would install it somewhere like c:\python364, then it is easier to find >>> than having it buried in its default installation path (I assume this is >>> Windows). >> >> >> python36 is a better choice of name > > > That's what I use; I assumed everyone else used the 3-digit version in the > path. Terrible assumption, given that it's not the default NOR a good idea :) > , as it doesn't cause problems when >> >> you upgrade to a new bugfix release. But there are permissions issues >> with dropping stuff straight into the root directory, which is why the >> default installers now put Python into Program Files. > > > It's not in the root, the executable will be: > > c:\python36\python.exe > > Not c:\python.exe. Yes, Windows allows you to have your own directories > within /. It's not Unix. I meant putting the pythonXY directory straight into the root. Yes, Windows allows it... but only if you are administrator. I think. Depends on the Windows version. And just FYI, Unix allows you to have your own directories within / too, so I don't know what your point is. Both OSes - recent versions, at least - restrict the creation of directories and files straight in the root. >> Jim, let the installer put it where it wants to, and make sure you've >> added it to PATH. Then you should be able to type "py" to start >> Python. > > > If I try to install 3.7 (as I already have 3.6) it suggests putting it in: > > c:\Users\users\AppData\Local\Programs\Python\Python37-32 > > Not actually a snappy path if you need to get there in a hurry to sort out > problems. That is, in a location 7 levels deep. > > Get it working anywhere first to find out what the problem is. > Get it working in the default location before you change things. The %PATH% environment variable exists to save you from typing seven levels of directory names. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Is this the right way to write a codec error handler?
On Sat, 20 Jan 2018 12:57:45 +0200, Serhiy Storchaka wrote:
> Just `end` instead of `end+1`.
Oops!
> And it is safer to use `bytes.decode(obj[start:end], 'latin1')` or
> `str(obj[start:end], 'latin1')` instead of
> `obj[start:end].decode('latin1')`. Just for the case if obj has
> overridden decode() method.
>
> Otherwise LGTM.
Thanks.
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: Can't get python running
On Sun, Jan 21, 2018 at 12:46 PM, Steven D'Aprano wrote: > On Sun, 21 Jan 2018 12:21:40 +1100, Chris Angelico wrote: > >> Jim, let the installer put it where it wants to, and make sure you've >> added it to PATH. Then you should be able to type "py" to start Python. > > Shouldn't the installer ensure that it puts "py" somewhere on the path? > Only if you tick the box to do that (or at least, that was the case a few versions ago). You don't have to do it manually, but you do have to make sure the box is ticked. (Which might just mean making sure you don't UNtick it; I don't know what the default is.) ChrisA -- https://mail.python.org/mailman/listinfo/python-list
