[issue27271] asyncio lost udp packets
New submission from valdemar pavesi:
hello,
I am using asyncio to handle 165 udp-packet per second
everything received by (datagram_received) will be put into a queue.
+++
124605 UDP packets send from client.
and received by server network:
dumpcap ( filter "port 5 and len > 100" )
Packets: 124605
correct number send and received. 124605
++
received by application:
def datagram_received(self, data, addr):
[2016-06-08 14:59:49] total udp = 124255,queue size =0
[2016-06-08 14:59:49] Got 124255 json report from server.
only 124255 received by application.
124605 - 124255 = 350 udp , received by network card , but application never
got it.
+++
code:
#
class UDPProtocolServerTraffica:
def __init__(self):
self.transport = None
# heart beat timer
self.HEARTBEAT_TIMEOUT = 10.0
self.REPORTSHOWTOTAL_TIMEOUT=60.0
self.MSG_UDP_HEARTBEAT =
b'\x01\x00\x00\x00\x11\x00\x00\x00\x01\x00\x00\x00\x00\x05\x00\x00\x00'
self.UDPCount = 0
def connection_made(self, transport):
self.transport = transport
# traffica startup message
self.transport.sendto(self.MSG_UDP_HEARTBEAT, (fns_remote_host,
fns_remote_port))
# start 10 seconds timeout timer
self.h_timeout =
asyncio.get_event_loop().call_later(self.HEARTBEAT_TIMEOUT,
self.timeout_heartbeat)
# show total report
self.h_timeout =
asyncio.get_event_loop().call_later(self.REPORTSHOWTOTAL_TIMEOUT,
self.timeout_report_showtotal)
def datagram_received(self, data, addr):
#fns_mmdu_ipaddr = addr [0]
#fns_mmdu_port = addr [1]
Report_Id = (int.from_bytes(data[0:2], byteorder='big'))
if Report_Id != 327:
self.UDPCount += 1
# send message to queue
asyncio_queue.put_nowait(data)
def pause_reading(self):
print('pause_reading')
def resume_reading(self):
print('resume_reading')
def error_received(self, exc):
print('Error received:', exc)
def connection_lost(self, exc):
print('stop', exc)
def timeout_heartbeat(self):
self.transport.sendto(self.MSG_UDP_HEARTBEAT, (fns_remote_host,
fns_remote_port))
self.h_timeout =
asyncio.get_event_loop().call_later(self.HEARTBEAT_TIMEOUT,
self.timeout_heartbeat)
#print('queue size ',asyncio_queue.qsize())
def timeout_report_showtotal(self):
self.h_timeout =
asyncio.get_event_loop().call_later(self.REPORTSHOWTOTAL_TIMEOUT,
self.timeout_report_showtotal)
self.displayReportTotalCount()
elasticsearch_get_all_reports()
def displayReportTotalCount(self):
logging.info('Total udp from fns: ' + str(self.UDPCount) + ' , queue
size: ' + str(asyncio_queue.qsize()) )
regards!
Valdemar
--
components: asyncio
messages: 267884
nosy: gvanrossum, haypo, valdemar.pavesi, yselivanov
priority: normal
severity: normal
status: open
title: asyncio lost udp packets
versions: Python 3.6
___
Python tracker
<http://bugs.python.org/issue27271>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27271] asyncio lost udp packets
valdemar pavesi added the comment: thanks Yury, I think we could lose inside the network, but we cannot lose inside of our application. regards! Valdemar -- ___ Python tracker <http://bugs.python.org/issue27271> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27271] asyncio lost udp packets
valdemar pavesi added the comment:
I am not getting any pause, or any message about buffer full.
def pause_reading(self):
print('pause_reading')
def resume_reading(self):
print('resume_reading')
and I cannot find a way to increase the receive buffer by asyncio.
--
___
Python tracker
<http://bugs.python.org/issue27271>
___
___
Python-bugs-list mailing list
Unsubscribe:
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27271] asyncio lost udp packets
valdemar pavesi added the comment: I do understand the possibility to lose on udp. and I am monitoring all machines involved. I am not losing on network, I am losing between our network-card-dumpcap and read udp from socket. is it possible that read will be blocked and not able to read ? during read and write to queue? I do have udp packets that will come in same microsecond. -- ___ Python tracker <http://bugs.python.org/issue27271> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27271] asyncio lost udp packets
valdemar pavesi added the comment: hi, I did made a change, removing the queue and calling corotine. and now lost udp is bigger. def datagram_received(self, data, addr): asyncio.ensure_future(process_report(data)) @asyncio.coroutine def process_report(data): tcmpdump got 38122 and asyncio got just 20711 2016-06-08 18:19:23,209 [INFO] 546 Total udp from fns: 20711 , queue size: 0 2016-06-08 18:19:23,209 [INFO] 417 Got 20711 json report from traffica server all udp received will be send by tcp to another server. (Got 20711 json report ) -- Added file: http://bugs.python.org/file43313/calling_corotine_wihout_queue_lost_bigger.png ___ Python tracker <http://bugs.python.org/issue27271> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27271] asyncio lost udp packets
valdemar pavesi added the comment: thanks Guido I will keep working with it. I am not able to debug between socket and asyncio-read-udp. There is no bottleneck on cpu/memory or network card. I cannot write debug with this heavy udp load. if I decrease the UDP per second then this problem go away. I know when it will start dropping and i never got any drop inside the network. ( tcpdump to validate it ). I will send it to python-tulip regards! Valdemar -- ___ Python tracker <http://bugs.python.org/issue27271> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue27271] asyncio lost udp packets
valdemar pavesi added the comment: thanks Guido and Yury I am new on python world. I was working with automation tests, sw implemented in Delphi in 199x. this year I got a python certification from University Texas Arlington University by EDX. and I already wrote 4 projects in python3 ,handling heavy traffic , one related to voice over lte simulation. thanks for all your contribution to the world without asking nothing back. regards! valdemar -- ___ Python tracker <http://bugs.python.org/issue27271> ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
