Hi, a better serialization system is definitely something I am beginnign to test. Another solution I am looking into is transforming the data I am storing into a more compact form, like replacing the strings I have with integers that represent them (as this portion of the app doesn't need to be human readable, it just needs to be used to make various decisions), and this will allow me to store the objects in the cache and get them out quickly without worring as much about serialization efficiency.
On Mon, May 20, 2013 at 7:43 PM, <[email protected]> wrote: > Send uWSGI mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of uWSGI digest..." > > > Today's Topics: > > 1. uwsgi emperor with multiple python versions (Jon Chappell) > 2. Re: uwsgi emperor with multiple python versions (Jon Chappell) > 3. Re: Django - SMTP (Venkatraman S) > 4. Re: sharing objects amongst workers (est) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Mon, 20 May 2013 19:16:54 -0400 > From: Jon Chappell <[email protected]> > To: [email protected] > Subject: [uWSGI] uwsgi emperor with multiple python versions > Message-ID: > < > cabvsgclha9hswbkgnnyndaypdezjxfsbrocdwvkjvdppxnw...@mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Hi, > > We are trying to run uwsgi with multiple python versions and are currently > compiling from source, but have also tried installation via pip. We're on > Ubuntu 10.04 which has Python 2.6 as the default Python version, but need > to run a mix of 2.6 and 2.7 applications that should be managed by the > emperor. The 2.6 application works fine, but the 2.7 application gets an > error when the emperor tries to start the workers. The Python 2.6 and 2.7 > plugins were both compiled also. > > The init script being used for uwsgi is: > description "uWSGI" > start on runlevel [2345] > stop on runlevel [06] > > respawn > > exec uwsgi --logto /var/log/uwsgi/emperor.log --emperor /etc/uwsgi/conf.d/ > > The ini file in question: > > [uwsgi] > > socket=/tmp/uwsgi.sock > chmod-socket=660 > pythonpath=/var/www/deployment > module=wsgi:application > master=True > pidfile=/tmp/deployment-master.pid > uid=uwsgi > gid=fm > harakiri=30 > max-requests=5000 > post-buffering=4096 > logto=/var/log/uwsgi/deployment.log > harakiri-verbose=false > buffer-size=32768 > virtualenv=/var/virtualenvs/deployment > plugin=python27 > stats=:1717 > processes=4 > > > > Here is the error: > > Python version: 2.6.5 (r265:79063, Oct 1 2012, 22:16:31) [GCC 4.4.3] > Set PythonHome to /var/virtualenvs/deployment > 'import site' failed; use -v for traceback > *** Python threads support is disabled. You can enable it with > --enable-threads *** > Python main interpreter initialized at 0xc55c50 > your server socket listen backlog is limited to 100 connections > your mercy for graceful operations on workers is 60 seconds > mapped 547800 bytes (534 KB) for 4 cores > *** Operational MODE: preforking *** > added /var/www/deployment/ to pythonpath. > Traceback (most recent call last): > File "/var/www/deployment/wsgi.py", line 2, in <module> > import os > ImportError: No module named os > unable to load app 0 (mountpoint='') (callable not found or import error) > > The virtualenv path has the correct python 2.7 binary installed in it, and > has all of the appropriate directory structure that a properly configured > virtualenv requires. Indeed, if the virtualenv is changed to use python > 2.6 (also removing the plugin from the ini file), uwsgi serves it up > correctly as well. > > -- > * > > Jon Chappell* > [email protected] > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.unbit.it/pipermail/uwsgi/attachments/20130520/47f0e6b1/attachment-0001.html > > > > ------------------------------ > > Message: 2 > Date: Mon, 20 May 2013 19:29:33 -0400 > From: Jon Chappell <[email protected]> > To: [email protected] > Subject: Re: [uWSGI] uwsgi emperor with multiple python versions > Message-ID: > <CABVsgcKKJgz_+vKawMrEo= > [email protected]> > Content-Type: text/plain; charset="iso-8859-1" > > Of course, no sooner did I send this than I finally figured out that the > python 2.7 plugin had been compiled wrong! I fixed that and it's now > working. > > Thanks! > > > On Mon, May 20, 2013 at 7:16 PM, Jon Chappell <[email protected]> wrote: > > > Hi, > > > > We are trying to run uwsgi with multiple python versions and are > currently > > compiling from source, but have also tried installation via pip. We're > on > > Ubuntu 10.04 which has Python 2.6 as the default Python version, but need > > to run a mix of 2.6 and 2.7 applications that should be managed by the > > emperor. The 2.6 application works fine, but the 2.7 application gets an > > error when the emperor tries to start the workers. The Python 2.6 and > 2.7 > > plugins were both compiled also. > > > > The init script being used for uwsgi is: > > description "uWSGI" > > start on runlevel [2345] > > stop on runlevel [06] > > > > respawn > > > > exec uwsgi --logto /var/log/uwsgi/emperor.log --emperor > /etc/uwsgi/conf.d/ > > > > The ini file in question: > > > > [uwsgi] > > > > socket=/tmp/uwsgi.sock > > chmod-socket=660 > > pythonpath=/var/www/deployment > > module=wsgi:application > > master=True > > pidfile=/tmp/deployment-master.pid > > uid=uwsgi > > gid=fm > > harakiri=30 > > max-requests=5000 > > post-buffering=4096 > > logto=/var/log/uwsgi/deployment.log > > harakiri-verbose=false > > buffer-size=32768 > > virtualenv=/var/virtualenvs/deployment > > plugin=python27 > > stats=:1717 > > processes=4 > > > > > > > > Here is the error: > > > > Python version: 2.6.5 (r265:79063, Oct 1 2012, 22:16:31) [GCC 4.4.3] > > Set PythonHome to /var/virtualenvs/deployment > > 'import site' failed; use -v for traceback > > *** Python threads support is disabled. You can enable it with > > --enable-threads *** > > Python main interpreter initialized at 0xc55c50 > > your server socket listen backlog is limited to 100 connections > > your mercy for graceful operations on workers is 60 seconds > > mapped 547800 bytes (534 KB) for 4 cores > > *** Operational MODE: preforking *** > > added /var/www/deployment/ to pythonpath. > > Traceback (most recent call last): > > File "/var/www/deployment/wsgi.py", line 2, in <module> > > import os > > ImportError: No module named os > > unable to load app 0 (mountpoint='') (callable not found or import error) > > > > The virtualenv path has the correct python 2.7 binary installed in it, > and > > has all of the appropriate directory structure that a properly configured > > virtualenv requires. Indeed, if the virtualenv is changed to use python > > 2.6 (also removing the plugin from the ini file), uwsgi serves it up > > correctly as well. > > > > -- > > * > > > > Jon Chappell* > > [email protected] > > > > > > -- > * > > Jon Chappell* > [email protected] > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.unbit.it/pipermail/uwsgi/attachments/20130520/ef787e61/attachment-0001.html > > > > ------------------------------ > > Message: 3 > Date: Tue, 21 May 2013 07:10:40 +0530 > From: Venkatraman S <[email protected]> > To: uWSGI developers and users list <[email protected]> > Subject: Re: [uWSGI] Django - SMTP > Message-ID: > < > can7tdfsc0qspc4ftq+7xrig59x8kjf8mp7koyqh7rucvvwe...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Should this be easier with spooler instead of celery? > > -V > @venkasub <http://about.me/venkasub> > > > > On Mon, May 20, 2013 at 10:39 PM, ?ukasz Mierzwa <[email protected] > >wrote: > > > It seems that it takes a lot of time to connect from django node to smtp > > server and --harakiri kills your worker. > > Verify with django "./manage.py shell" that you can send those emails > with > > your current code. > > After that I would suggest to move smtp handling to background queue > using > > celery and https://pypi.python.org/pypi/django-celery-email > > > > > > 2013/5/20 Daniel Kopka <[email protected]> > > > >> Hi, > >> > >> When I try to send email from Django 1.2 via uWSGI in log I receives > >> (uwsgi 1.9.10): > >> > >> uwsgi_response_write_body_do(): Broken pipe [core/writer.c line 248] > >> IOError: write error > >> *** HARAKIRI ON WORKER 6 (pid: 30863, try: 1) *** > >> HARAKIRI: -- syscall> 42 0x8 0x7fff3d23f710 0x10 0xe 0x1 0x100 > >> 0x7fff3d23f6b0 0x7fe481bcb8ad > >> HARAKIRI: -- wchan> inet_stream_connect > >> *** backtrace of 30863 *** > >> xx_uWSGI worker 6(uwsgi_backtrace+0x29) [0x451b49] > >> xx_uWSGI worker 6(what_i_am_doing+0x17) [0x451fc7] > >> /lib64/libc.so.6(+0x32920) [0x7fe480095920] > >> /lib64/libpthread.so.0(__connect+0x2b) [0x7fe481bcb8ab] > >> /srv/venvs/xx/lib64/python2.6/lib-dynload/_socketmodule.so(+0x6018) > >> [0x7fe47de2c018] > >> /srv/venvs/xx/lib64/python2.6/lib-dynload/_socketmodule.so(+0x809c) > >> [0x7fe47de2e09c] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x4e26) > [0x7fe4807025c6] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927) [0x7fe480704657] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304) > [0x7fe480702aa4] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927) [0x7fe480704657] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304) > [0x7fe480702aa4] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef) > [0x7fe480703b8f] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927) [0x7fe480704657] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x5304) > [0x7fe480702aa4] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalCodeEx+0x927) [0x7fe480704657] > >> /usr/lib64/libpython2.6.so.1.0(+0x6adad) [0x7fe480697dad] > >> /usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53) [0x7fe480670c63] > >> /usr/lib64/libpython2.6.so.1.0(+0x566af) [0x7fe4806836af] > >> /usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53) [0x7fe480670c63] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_CallObjectWithKeywords+0x43) > >> [0x7fe4806fcc93] > >> /usr/lib64/libpython2.6.so.1.0(PyInstance_New+0x76) [0x7fe480682e06] > >> /usr/lib64/libpython2.6.so.1.0(PyObject_Call+0x53) [0x7fe480670c63] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x47d4) > [0x7fe480701f74] > >> /usr/lib64/libpython2.6.so.1.0(PyEval_EvalFrameEx+0x63ef) > [0x7fe480703b8f] > >> *** end of backtrace *** > >> HARAKIRI: --- uWSGI worker 6 (pid: 30863) WAS managing request > >> /accounts/register/ since Mon May 20 18:53:54 2013 --- > >> *** HARAKIRI ON WORKER 6 (pid: 30863, try: 2) *** > >> HARAKIRI: -- wchan> do_exit > >> DAMN ! worker 6 (pid: 30863) died, killed by signal 9 :( trying respawn > >> ... > >> Respawned uWSGI worker 6 (new pid: 30999) > >> > >> > >> Any ideas what is wrong or where to look for the error? > >> > >> > >> > >> -- > >> Pozdrawiam, > >> Daniel Kopka > >> Tel: 502276071 > >> > >> _______________________________________________ > >> uWSGI mailing list > >> [email protected] > >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > >> > >> > > > > > > -- > > ?ukasz Mierzwa > > > > _______________________________________________ > > uWSGI mailing list > > [email protected] > > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > > > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.unbit.it/pipermail/uwsgi/attachments/20130521/a4267cf4/attachment-0001.html > > > > ------------------------------ > > Message: 4 > Date: Tue, 21 May 2013 10:45:09 +0800 > From: est <[email protected]> > To: uWSGI developers and users list <[email protected]> > Subject: Re: [uWSGI] sharing objects amongst workers > Message-ID: > <CA+= > [email protected]> > Content-Type: text/plain; charset="utf-8" > > > No, you cannot share mongodb connections, you need at least one > connection > per-worker. > > Actually, technically you can. > > For posix only cross-worker connection sharing, use sendmsg() to pass the > MongoDB connection fd from one worker to another. > > Only if the uWSGI master supports sendmsg() between workers and mules > > Just an idea. > > > > > > On Sat, May 18, 2013 at 1:44 PM, Roberto De Ioris <[email protected]> > wrote: > > > > > > We are using the pymongo driver with mongodb and as it stands, it looks > > > like each worker gets its own connection. Is there any way to share a > > > database connection in memory among the workers? I've read about ksm > on > > > the readthedocs page, is that applicable here? > > > > No, you cannot share mongodb connections, you need at least one > connection > > per-worker. If you are in multithread you can use connection pools (i do > > not know if they are supported by pymongo) > > > > > > > > We also have a bunch of python objects being stored in memory, would > KSM > > > be > > > helpful in sharing memory among the workers for this? > > > > > > > > For example: > > > Currently we have an identical list being stored on each worker (we > keep > > > the list stored in a python variable because its large and we need to > > > access it quickly so storing in the uwsgi cache and pickling and > > > unpickling > > > (the only way I can think of to store a python object in the cache) > each > > > request, would be too slow) and we have 6 workers. This means we are > > > using > > > several times the memory we actually need to. Is there any way to > share > > > a > > > global python object among all of the workers? All we do is read from > > it, > > > we only modify it when we are completely refreshing it periodically. > > > > > > The overarching question is, how do i share db connections and python > > > objects globally, so each worker doesn't need to have their own copy of > > > the > > > > You will hardly find something faster than the uWSGI cache (there are > > quite astonishing benchmark of it over memcached and redis), so i would > > focus on finding a better serialization system. Can't you represent it in > > a more binary-friendly way without using pickle/marshalling ? > > > > In the ruby world the approach is having a server storing the object and > > using a specialized framework for remotely manipulating it (drb). I do > not > > know if there is something similar in python. > > > > > > -- > > Roberto De Ioris > > http://unbit.it > > _______________________________________________ > > uWSGI mailing list > > [email protected] > > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > > > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.unbit.it/pipermail/uwsgi/attachments/20130521/b3020af9/attachment.html > > > > ------------------------------ > > _______________________________________________ > uWSGI mailing list > [email protected] > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi > > > End of uWSGI Digest, Vol 44, Issue 29 > ************************************* >
_______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
