Handling of disconnecting clients in asyncio

[Johannes Bauer]

Hi group,

I'm trying to get into async programming using Python. Concretely I open
a UNIX socket server in my application. The UNIX socket server generates
events and also receives commands/responds to them.

I do this by:

async def _create_local_server(self):
await asyncio.start_unix_server(self._local_server_tasks, path = "foo"))

And then gather the command/response and event tasks:

async def _local_server_tasks(self, reader, writer):
await asyncio.gather(
self._local_server_commands(reader, writer),
self._local_server_events(reader, writer),
)

I believe so far this is okay, right? If not, please tell me. Anyways,
the event loop as an example:

async def _local_server_events(self, reader, writer):
while True:
await asyncio.sleep(1)
writer.write(b"event\n")

And the command/response loop, obviously simplified:

async def _local_server_commands(self, reader, writer):
while True:
msg = await reader.readline()
writer.write(msg)

Now I'm having the following issue: A client connects to my server and
then properly disconnects (shutdown/RDWR, close). This causes the await
reader.readline() to return an empty message (after which I can properly
end the _local_server_commands loop).

However, the _local_server_events loop does get no such notificiation.
Nor does writer.write() throw an exception that I could catch (and exit
as a consequence). Instead, I get this on stderr:

socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
socket.send() raised exception.
[...]

My questions are:

1. Is the design generally sane or is this usually done differently?
I.e., am I making any obvious beginner mistakes?

2. What is the proper way of discovering a peer has disconnected and
exiting cleanly?

Thanks in advance,
All the best,
Johannes

-- 
"Performance ist nicht das Problem, es läuft ja nachher beides auf der
selben Hardware." -- Hans-Peter Diettrich in d.s.e.
-- 
https://mail.python.org/mailman/listinfo/python-list

Exception

[ast]

Hi

It is not clear to me why the following code
generates 2 exceptions, ZeroDivisionError and
ArithmeticError. Since ZeroDivisionError is
catched, it shoud not bubble out.

Found here:
https://www.pythonsheets.com/notes/python-new-py3.html

>>> def func():
... try:
... 1 / 0
... except ZeroDivisionError:
... raise ArithmeticError
...
>>> func()
Traceback (most recent call last):
  File "", line 3, in func
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "", line 1, in 
  File "", line 5, in func
ArithmeticError


There is a work around (python >= 3.3)

>>> def func():
... try:
... 1 / 0
... except ZeroDivisionError:
... raise ArithmeticError from None
...
>>> func()
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 5, in func
ArithmeticError
--
https://mail.python.org/mailman/listinfo/python-list

RE: Exception

[David Raymond]

I believe the idea is that previously, if you were handling an exception, and 
your handling code caused its own exception, then you would get no info about 
the original exception. So all useful information about the original problem 
would get lost because of an oopsie in the handling code. So it got changed to 
show both the original exception as well as the final one to be more 
useful/helpful.

The raise ... from None is basically you explicitly saying "I am 
 raising a different exception, so don't worry about the old one"

Or such is my limited understanding anyway.


-Original Message-
From: Python-list  On 
Behalf Of ast
Sent: Tuesday, September 24, 2019 9:04 AM
To: python-list@python.org
Subject: Exception

Hi

It is not clear to me why the following code
generates 2 exceptions, ZeroDivisionError and
ArithmeticError. Since ZeroDivisionError is
catched, it shoud not bubble out.

Found here:
https://www.pythonsheets.com/notes/python-new-py3.html

 >>> def func():
... try:
... 1 / 0
... except ZeroDivisionError:
... raise ArithmeticError
...
 >>> func()
Traceback (most recent call last):
   File "", line 3, in func
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "", line 1, in 
   File "", line 5, in func
ArithmeticError


There is a work around (python >= 3.3)

 >>> def func():
... try:
... 1 / 0
... except ZeroDivisionError:
... raise ArithmeticError from None
...
 >>> func()
Traceback (most recent call last):
   File "", line 1, in 
   File "", line 5, in func
ArithmeticError
-- 
https://mail.python.org/mailman/listinfo/python-list
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exception

[אורי]

https://stackoverflow.com/a/24752607/1412564
אורי
u...@speedy.net


On Tue, Sep 24, 2019 at 4:07 PM ast  wrote:

> Hi
>
> It is not clear to me why the following code
> generates 2 exceptions, ZeroDivisionError and
> ArithmeticError. Since ZeroDivisionError is
> catched, it shoud not bubble out.
>
> Found here:
> https://www.pythonsheets.com/notes/python-new-py3.html
>
>  >>> def func():
> ... try:
> ... 1 / 0
> ... except ZeroDivisionError:
> ... raise ArithmeticError
> ...
>  >>> func()
> Traceback (most recent call last):
>File "", line 3, in func
> ZeroDivisionError: division by zero
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
>File "", line 1, in 
>File "", line 5, in func
> ArithmeticError
>
>
> There is a work around (python >= 3.3)
>
>  >>> def func():
> ... try:
> ... 1 / 0
> ... except ZeroDivisionError:
> ... raise ArithmeticError from None
> ...
>  >>> func()
> Traceback (most recent call last):
>File "", line 1, in 
>File "", line 5, in func
> ArithmeticError
> --
> https://mail.python.org/mailman/listinfo/python-list
>
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Exception

[ast]

Le 24/09/2019 à 15:51, אורי a écrit :

https://stackoverflow.com/a/24752607/1412564


thank you for link. it's clear now
--
https://mail.python.org/mailman/listinfo/python-list

Tahoe-LAFS on Python 3 - Call for Porters

[Jean-Paul Calderone]

Hello Pythonistas,


Earlier this year a number of Tahoe-LAFS
 community members began an effort
to port Tahoe-LAFS from Python 2 to Python 3.  Around five people are
currently involved in a part-time capacity.  We wish to accelerate the
effort to ensure a Python 3-compatible release of Tahoe-LAFS can be made
before the end of upstream support for CPython 2.x.


Tahoe-LAFS is a Free and Open system for private, secure, decentralized
storage.  It encrypts and distributes your data across multiple servers.
If some of the servers fail or are taken over by an attacker, the entire
file store continues to function correctly, preserving your privacy and
security.


Foolscap , a dependency of Tahoe-LAFS,
is also being ported.  Foolscap is an object-capability-based RPC protocol
with flexible serialization.


Some details of the porting effort are available in a milestone on the
Tahoe-LAFS trac instance
.


For this help, we are hoping to find a person/people with significant prior
Python 3 porting experience and, preferably, some familiarity with Twisted,
though in general the Tahoe-LAFS project welcomes contributors of all
backgrounds and skill levels.


We would prefer someone to start with us as soon as possible and no later
than October 15th. If you are interested in this opportunity, please send
us any questions you have, as well as details of your availability and any
related work you have done previously (GitHub, LinkedIn links, etc). If you
would like to find out more about this opportunity, please contact us at
jessielisbetfrance at gmail (dot) com or on IRC in #tahoe-lafs on Freenode.

Jean-Paul
-- 
https://mail.python.org/mailman/listinfo/python-list

CSV reader ignore brackets

[Mihir Kothari]

Hi Team,

I am using python 3.4. I have a CSV file as below:

ABC,PQR,(TEST1,TEST2)
FQW,RTE,MDE

Basically comma-separated rows, where some rows have a data in column which
is array like i.e. in brackets.
So I need to read the file and treat such columns as one i.e. do not
separate based on comma if it is inside the bracket.

In short I need to read a CSV file where separator inside the brackets
needs to be ignored.

Output:
Column:   1   23
Row1:ABC  PQR  (TEST1,TEST2)
Row2:FQW  RTE  MDE

Can you please help with the snippet?

Regards,
Mihir.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV reader ignore brackets

[Cameron Simpson]

On 24Sep2019 15:55, Mihir Kothari  wrote:

I am using python 3.4. I have a CSV file as below:

ABC,PQR,(TEST1,TEST2)
FQW,RTE,MDE


Really? No quotes around the (TEST1,TEST2) column value? I would have 
said this is invalid data, but that does not help you.



Basically comma-separated rows, where some rows have a data in column which
is array like i.e. in brackets.
So I need to read the file and treat such columns as one i.e. do not
separate based on comma if it is inside the bracket.

In short I need to read a CSV file where separator inside the brackets
needs to be ignored.

Output:
Column:   1   23
Row1:ABC  PQR  (TEST1,TEST2)
Row2:FQW  RTE  MDE

Can you please help with the snippet?


I would be reaching for a regular expression. If you partition your 
values into 2 types: those starting and ending in a bracket, and those 
not, you could write a regular expression for the former:


   \([^)]*\)

which matches a string like (.) (with, importantly, no embedded 
brackets, only those at the beginning and end.


And you can write a regular expression like:

   [^,]*

for a value containing no commas i.e. all the other values.

Test the bracketed one first, because the second one always matches  
something.


Then you would not use the CSV module (which expects better formed data 
than you have) and instead write a simple parser for a line of text 
which tries to match one of these two expressions repeatedly to consume 
the line. Something like this (UNTESTED):


   bracketed_re = re.compile(r'\([^)]*\)')
   no_commas_re = re.compile(r'[^,]*')

   def split_line(line):
 line = line.rstrip()  # drop trailing whitespace/newline
 fields = []
 offset = 0
 while offset < len(line):
   m = bracketed_re.match(line, offset)
   if m:
 field = m.group()
   else:
 m = no_commas_re.match(line, offset)   # this always matches
 field = m.group()
   fields.append(field)
   offset += len(field)
   if line.startswith(',', offset):
 # another column
 offset += 1
   elif offset < len(line):
 raise ValueError(
   "incomplete parse at offset %d, line=%r" % (offset, line))
 return fields

Then read the lines of the file and split them into fields:

   row = []
   with open(datafilename) as f:
 for line in f:
   fields = split_line(line)
   rows.append(fields)

So basicly you're writing a little parser. If you have nested brackets 
things get harder.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Re: CSV reader ignore brackets

[MRAB]

On 2019-09-25 00:09, Cameron Simpson wrote:

On 24Sep2019 15:55, Mihir Kothari  wrote:

I am using python 3.4. I have a CSV file as below:

ABC,PQR,(TEST1,TEST2)
FQW,RTE,MDE


Really? No quotes around the (TEST1,TEST2) column value? I would have
said this is invalid data, but that does not help you.


Basically comma-separated rows, where some rows have a data in column which
is array like i.e. in brackets.
So I need to read the file and treat such columns as one i.e. do not
separate based on comma if it is inside the bracket.

In short I need to read a CSV file where separator inside the brackets
needs to be ignored.

Output:
Column:   1   23
Row1:ABC  PQR  (TEST1,TEST2)
Row2:FQW  RTE  MDE

Can you please help with the snippet?


I would be reaching for a regular expression. If you partition your
values into 2 types: those starting and ending in a bracket, and those
not, you could write a regular expression for the former:

 \([^)]*\)

which matches a string like (.) (with, importantly, no embedded
brackets, only those at the beginning and end.

And you can write a regular expression like:

 [^,]*

for a value containing no commas i.e. all the other values.

Test the bracketed one first, because the second one always matches
something.

Then you would not use the CSV module (which expects better formed data
than you have) and instead write a simple parser for a line of text
which tries to match one of these two expressions repeatedly to consume
the line. Something like this (UNTESTED):

 bracketed_re = re.compile(r'\([^)]*\)')
 no_commas_re = re.compile(r'[^,]*')

 def split_line(line):
   line = line.rstrip()  # drop trailing whitespace/newline
   fields = []
   offset = 0
   while offset < len(line):
 m = bracketed_re.match(line, offset)
 if m:
   field = m.group()
 else:
   m = no_commas_re.match(line, offset)   # this always matches
   field = m.group()
 fields.append(field)
 offset += len(field)
 if line.startswith(',', offset):
   # another column
   offset += 1
 elif offset < len(line):
   raise ValueError(
 "incomplete parse at offset %d, line=%r" % (offset, line))
   return fields

Then read the lines of the file and split them into fields:

 row = []
 with open(datafilename) as f:
   for line in f:
 fields = split_line(line)
 rows.append(fields)

So basicly you're writing a little parser. If you have nested brackets
things get harder.


You can simplify that somewhat to this:

import re
rows = []

with open(datafilename) as f:
for line in f:
rows.append(re.findall(r'(\([^)]*\)|(?=.)[^,\n]*),?', line))
--
https://mail.python.org/mailman/listinfo/python-list

Re: CSV reader ignore brackets

[Skip Montanaro]

How about just replacing *\(([^)]*)\)* with *"\1"* in a wrapper class's
line reading method? (I think I have the re syntax approximately right.)
The csv reader will "just work". Again, nesting parens not allowed.

Skip
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: CSV reader ignore brackets

[Cameron Simpson]

On 24Sep2019 19:02, Skip Montanaro  wrote:

How about just replacing *\(([^)]*)\)* with *"\1"* in a wrapper class's
line reading method?


Will that work if the OP's (TEST1,TEST2) term itself contains quotes?  
Not that his example data did, but example data are usually incomplete 
:-)


Also, that would match FOO(TEST1,TEST2)BAH as well (making 
FOO"(TEST1,TEST2)"BAH.  Which might be wanted, or be not wanted or be 
bad data (including but not restricted to csv module unparsable data).  
I was deliberately being very conservative and kind of treating brackets 
like quotes (needest at start and end) but not trying to hit things in 
one go.  Better to match exactly the special case you expect and then 
scour of mismatches than to incorrectly match and have that mistake 
buried in the data.



(I think I have the re syntax approximately right.)
The csv reader will "just work". Again, nesting parens not allowed.


Otherwise, a neat idea.

Besides, the point isn't the shortest code but to illustrate the 
idea of handling special syntax.


Cheers,
Cameron Simpson 
--
https://mail.python.org/mailman/listinfo/python-list

Python in The Economist

[Frank Millman]

The latest Technology Quarterly in The Economist is about "The Internet 
Of Things".


Python gets a mention in an article on "How to build a disposable 
microchip". It is quite a long article, so here are the relevant extracts.


"The goal is to produce a robust, bendable, mass-producible computer, 
complete with sensors and the ability to communicate with the outside 
world, for less than $0.01 apiece. A prototype version, shown off at 
Arm's headquarters in Cambridge, looks like a stiffer-than-usual piece 
of tape festooned with circuit traces."


"The chip uses a simple form of machine learning called a Bayesian 
classifier. Flexibility of use was sacrificed: to keep thinks as cheap 
and simple as possible the algorithm is etched directly into the 
plastic, meaning the chips are not reprogrammable."


"Since chip design is expensive, and chip designers scarce, he and his 
team have been working on software tools to simplify that task. The idea 
is to describe a new algorithm in Python, a widely used programming 
language, and then have software turn it into a circuit diagram that can 
be fed into Pragmatic's chipmaking machines. That approach has attracted 
interest from DARPA ..."


Hope this is of interest.

Frank Millman

--
https://mail.python.org/mailman/listinfo/python-list