[issue40352] SocketHandler silently drops log messages on re-connect

[Oleg Nykolyn]

New submission from Oleg Nykolyn :

Hi,

I've faced this issue when using logging.handlers.SocketHandler AWS TCP 
balancer. AWS balancer uses 60 second time-out by default (max 4000s), thus 
resulting in lots of closed sockets during inactive periods.
SocketHandler.send() drops current message on any socket errors, so only next 
message gets logged.
I've tried to reproduce this using Lib unit tests, but didn't find any easy way 
to close() socket on test server side.
 
Sample client/server scripts attached, server output:

Got connection from:  ('127.0.0.1', 58044)
Got message:  b'Message #1\n'
Got message:  b'Message #2\n'
Got connection from:  ('127.0.0.1', 58045)
Got message:  b'Message #5\n'

Server closes incoming connection is 2 seconds, client looses messages #3 and 
#4.

--
components: Library (Lib)
files: Archive.zip
messages: 366920
nosy: Oleg Nykolyn
priority: normal
severity: normal
status: open
title: SocketHandler silently drops log messages on re-connect
type: behavior
versions: Python 3.8
Added file: https://bugs.python.org/file49080/Archive.zip

___
Python tracker 
<https://bugs.python.org/issue40352>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue40352] SocketHandler silently drops log messages on re-connect

[Oleg Nykolyn]

Oleg Nykolyn  added the comment:

There are multiple servers running in Kubrnetes cluster - API servers based on 
Django, celery workers, etc. All of them send logs to AWS TCP balancer, which 
acts as balancer for vector service[1], which send logs to Elasticsearch.
Basically we have following logging pipeline: python-based services -> AWS TCP 
network balancer -> vector -> Elasticsearch.
AWS network balancer has an option called "Idle timeout" with max value of 3600 
seconds[2].
Log messages are logged successfully at first, but fail(one message gets lost 
on re-connect) if there is gap between messages, corresponding to "Idle 
timeout".

1: https://github.com/timberio/vector
2: 
https://docs.aws.amazon.com/elasticloadbalancing/latest/network/network-load-balancers.html#connection-idle-timeout

--

___
Python tracker 
<https://bugs.python.org/issue40352>
___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com