** Description changed: [Impact] Applications using package python-tornado v5.1.1 or earlier are susceptible to an out of memory error related to websockets. [Other Info] Upstream commit(s): https://github.com/tornadoweb/tornado/pull/2351/commits/20becca336caae61cd24f7afba0e177c0a210c70 $ git remote -v origin https://github.com/tornadoweb/tornado.git (fetch) origin https://github.com/tornadoweb/tornado.git (push) $ git describe --contains 20becca3 v5.1.0b1~28^2~1 $ rmadison python3-tornardo => python3-tornado | 4.2.1-1ubuntu3 | xenial python3-tornado | 4.5.3-1 | bionic/universe => python3-tornado | 4.5.3-1ubuntu0.1 | bionic-updates/universe python3-tornado | 6.0.3+really5.1.1-3 | focal/universe python3-tornado | 6.0.4-2 | groovy/universe python3-tornado | 6.0.4-3 | hirsute/universe python3-tornado | 6.1.0-1 | hirsute-proposed/universe [Original Description] Tornado has no 'flow control' for websockets. A websocket will receive data as fast as it can, and store the data in a deque. If that data is not consumed as fast as it is written, then that deque will grow in size indefinitely, ultimately leading to a memory error and killing the process. Fix is to use a Queue. Read and get messages from the queue on the client side. Patch file [0] Commit history [1] GitHub [2] Issue [3] [0] https://patch-diff.githubusercontent.com/raw/tornadoweb/tornado/pull/2351.patch [1] https://github.com/tornadoweb/tornado/pull/2351/commits [2] https://github.com/tornadoweb/tornado [3] https://github.com/tornadoweb/tornado/issues/2341 [Test Case] - I will be attaching two python test files. - client.py + I will be attaching two python test files. + client.py server.py # create lxc container & limits on memory and turn off swap $ sudo apt-get install lxc lxd $ lxd init - $ lxc launch ubuntu:18.04 bionic-python-tornado + $ lxc launch ubuntu:18.04 bionic-python-tornado # shrink server size lxc config set server limits.cpu 2 # changes ram setting lxc config set server limits.memory 150MB # severely limits amount of swap used [4] lxc config set bionic-py-tornado limits.memory.swap false # install dev tools and download source code - $ lxc exec bionic-python-tornado bash + $ lxc exec bionic-python-tornado bash $ apt-get update $ apt install ubuntu-dev-tools -y - $ pull-lp-source python-tornado bionic + $ pull-lp-source python-tornado bionic $ sudo apt build-dep . - # copy client.py and server.py to + # copy client.py and server.py to # $ ~/python-tornado-4.5.3/demos $ scp or touch client.py and server.py # build code - $ python3 setup.py build - $ python3 setup.py install + $ python3 setup.py build + $ python3 setup.py install - # I have 3 terminals open + # I have 3 terminals open 2 for executing python, one for the client and one for server - and another one using top to view memory constraints + and another one using top to view memory constraints # run server.py, client.py, and top in separate terminals $ python3 demos/client.py $ python3 demos/server.py - $ top + $ top What gets print out in the client.py is the length of the - collections.deqeue + collections.deque - - In the server.py prints out messages like: + In the server.py prints out messages like: message: keep alive * press ctrl+E for showing memory in MB in the terminal with top top - shows that swap is off/ running very low and our memory is only 150MB - Although I never hit the oom exception that is expected to be thrown, - you can check dmesg + Although I never hit the oom exception that is expected to be thrown, + you can check dmesg $ sudo dmesg | grep -i python - looks similar to this: + looks similar to this: [ 3250.067833] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=lxc.payload.iptest,mems_allowed=0,oom_memcg=/lxc.payload.iptest,task_memcg=/lxc.payload.iptest,task=python3,pid=44889,uid=1000000 [ 3250.067838] Memory cgroup out of memory: Killed process 44889 (python3) total-vm:304616kB, anon-rss:235152kB, file-rss:0kB, shmem-rss:0kB, UID:1000000 pgtables:628kB oom_score_adj:0 [ 3250.075096] oom_reaper: reaped process 44889 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0k - After either adding the patch or running focal or later versions - *pull-lp-source python-tornado focal + After either adding the patch or running focal or later versions + *pull-lp-source python-tornado focal We can run the exact same setup again, and this time it shows that the - new queue object only has a length of 1. + new queue object has only a length of 1. [4] https://discuss.linuxcontainers.org/t/limiting-swap-in-lxc-container/6343/4 [5] attached screenshot
** Attachment added: "client.py" https://bugs.launchpad.net/ubuntu/xenial/+source/python-tornado/+bug/1903733/+attachment/5451896/+files/client.py -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1903733 Title: Out of memory issue for websocket client To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-tornado/+bug/1903733/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
