Virtualenv loses context
I am somewhat new to Python and virtualenv. I have setup a virtualenv with no site packages under the assumption that all 3rd party packages will be installed into the site-packages directory for the virtualenv. Another assumption I have about virtualenv is that it's completely self contained. When I first setup the virtualenv, install my packages and then install my application both of those assumptions seem to be correct. However, at some point I start getting ImportErrors. The be clear, I tested the application and it was all working correctly, the ImportErrors seem to show up at some random point that I cannot reproduce. I have a requirements file that I made with 'pip freeze'. I try to use this to reinstall the packages but I get errors saying that I cannot install to the location. The location is the global site-packages directory on my system. I have the virtualenv activated while installing so I dont know why this is happening. Due to something about Centos6 I cant install M2Crypto using pip and have to do it manually. This gives me the same types of errors: (botnet_etl)[swright@localhost M2Crypto]$ ./fedora_setup.sh install running install error: can't create or remove files in install directory The following error occurred while trying to add or remove files in the installation directory: [Errno 13] Permission denied: '/usr/lib64/python2.6/site-packages/test-easy-inst all-29133.write-test' The installation directory you specified (via --install-dir, --prefix, or the distutils default setting) was: /usr/lib64/python2.6/site-packages/ Perhaps your account does not have write access to this directory? If the installation directory is a system-owned directory, you may need to sign in as the administrator or "root" account. If you do not have administrative access to this machine, you may wish to choose a different installation directory, preferably one that is listed in your PYTHONPATH environment variable. For information on other options, you may wish to consult the documentation at: http://peak.telecommunity.com/EasyInstall.html Please make the appropriate changes for your system and try again. The only fix I have managed for this problem so far is to reinstall the entire virtualenv from scratch. This works fine for a while and I have to repeat the process. I would think that I was doing something wrong if the ImportErrors showed up every time I attempt to run the app but the fact that they show up later leads me to believe something funky is going on. Os = Centos6 Python = 2.6.6 Thanks Steve -- http://mail.python.org/mailman/listinfo/python-list
Re: Virtualenv loses context
This may have something to do with it. I create the virtualenv in: /home/swright/workspace/botnet_etl/ After I install my app the directory structure looks like this: (botnet_etl)[swright@localhost botnet_etl]$ ll total 32 drwxrwxr-x 4 swright swright 4096 Dec 19 09:21 app drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 bin drwxrwxr-x 9 swright swright 4096 Dec 18 10:24 build drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 include drwxrwxr-x 4 swright swright 4096 Dec 18 10:34 lib -rw-rw-r-- 1 swright swright 1973 Dec 17 15:35 README -rw-rw-r-- 1 swright swright 109 Dec 17 15:35 stable-req.txt drwxrwxr-x 3 swright swright 4096 Dec 17 15:35 utils When I run the following command the path is not showing up which is probably what is causing the problem. I have to reinstall from scratch to see if this is different at install time. But, am I supposed to create virtualenv in specific location in the file system? That seems a little odd. -- http://mail.python.org/mailman/listinfo/python-list
Re: Virtualenv loses context
I wrote this little test script to prove my virtualenv is reading from global site packages: Script Content from distutils.sysconfig import get_python_lib print (get_python_lib()) Script Output (botnet_etl)[swright@localhost app]$ python test.py /usr/lib/python2.6/site-packages (botnet_etl)[swright@localhost app]$ Please note that (botnet_etl) is the name of the virtualenv -- http://mail.python.org/mailman/listinfo/python-list
Re: Virtualenv loses context
This may have something to do with it. I create the virtualenv in: /home/swright/workspace/botnet_etl/ After I install my app the directory structure looks like this: (botnet_etl)[swright@localhost botnet_etl]$ ll total 32 drwxrwxr-x 4 swright swright 4096 Dec 19 09:21 app drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 bin drwxrwxr-x 9 swright swright 4096 Dec 18 10:24 build drwxrwxr-x 3 swright swright 4096 Dec 18 10:34 include drwxrwxr-x 4 swright swright 4096 Dec 18 10:34 lib -rw-rw-r-- 1 swright swright 1973 Dec 17 15:35 README -rw-rw-r-- 1 swright swright 109 Dec 17 15:35 stable-req.txt drwxrwxr-x 3 swright swright 4096 Dec 17 15:35 utils When I run the following command the path is not showing up which is probably what is causing the problem. Clearly the site packages for my virtualenv are not listed. I have to reinstall from scratch to see if this is different at install time. (botnet_etl)[swright@localhost app]$ python -m site sys.path = [ '/home/swright/workspace/botnet_etl/app', '/usr/lib/python2.6/site-packages/pymongo-2.3-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', ] USER_BASE: '/home/swright/.local' (exists) USER_SITE: '/home/swright/.local/lib/python2.6/site-packages' (doesn't exist) ENABLE_USER_SITE: True But, am I supposed to create virtualenv in specific location in the file system? That seems a little odd -- http://mail.python.org/mailman/listinfo/python-list
Re: Virtualenv loses context
Just installed a brand new virtualenv along with two packages. Ran this and I got nothing: (venvtest)[swright@localhost venvtest]$ python -m site (venvtest)[swright@localhost venvtest]$ I expected to have at least one path in sys.path [swright@localhost workspace]$ virtualenv --no-site-packages venvtest New python executable in venvtest/bin/python Installing setuptoolsdone. Installing pip...done. [swright@localhost venvtest]$ source bin/activate (venvtest)[swright@localhost venvtest]$ python -m site (venvtest)[swright@localhost venvtest]$ (venvtest)[swright@localhost venvtest]$ pip install -r stable-req.txt Downloading/unpacking pycrypto==2.6 (from -r stable-req.txt (line 1)) Downloading pycrypto-2.6.tar.gz (443kB): 443kB downloaded Running setup.py egg_info for package pycrypto Downloading/unpacking pymongo==2.4 (from -r stable-req.txt (line 2)) Downloading pymongo-2.4.tar.gz (273kB): 273kB downloaded Running setup.py egg_info for package pymongo Downloading/unpacking yolk==0.4.3 (from -r stable-req.txt (line 3)) Downloading yolk-0.4.3.tar.gz (86kB): 86kB downloaded Running setup.py egg_info for package yolk warning: no files found matching '*.txt' under directory 'tests' warning: no files found matching '*.conf' under directory 'docs' warning: no files found matching '*.css_t' under directory 'docs' warning: no files found matching 'indexsidebar.html' under directory 'docs' warning: no files found matching 'tests/test_cli.py' Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python 2.6/site-packages/setuptools-0.6c11-py2.6.egg (from yolk==0.4.3->-r stable-req.txt ( line 3)) ... lots more text... (venvtest)[swright@localhost venvtest]$ pwd /home/swright/workspace/venvtest (venvtest)[swright@localhost venvtest]$ ll lib/python2.6/site-packages/ total 372 drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 bson drwxrwxr-x 10 swright swright 4096 Dec 19 09:34 Crypto -rw-rw-r-- 1 swright swright237 Dec 19 09:32 easy-install.pth drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 gridfs drwxrwxr-x 4 swright swright 4096 Dec 19 09:32 pip-1.2.1-py2.6.egg drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pycrypto-2.6-py2.6.egg-info drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pymongo drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 pymongo-2.4-py2.6.egg-info -rw-r--r-- 1 swright swright 333447 Dec 5 09:13 setuptools-0.6c11-py2.6.egg -rw-rw-r-- 1 swright swright 30 Dec 19 09:32 setuptools.pth drwxrwxr-x 3 swright swright 4096 Dec 19 09:34 yolk drwxrwxr-x 2 swright swright 4096 Dec 19 09:34 yolk-0.4.3-py2.6.egg-info (venvtest)[swright@localhost venvtest]$ -- http://mail.python.org/mailman/listinfo/python-list
Re: Virtualenv loses context
So I reinstalled the virtualenv for my project from scratch. Application runs as expected. Here are the notes that I took while installing. Interestingly the command 'python -m site' produces no output now. Notice that before I reinstalled the virtualenv I got a bunch of paths from that command. I am going to reboot my machine to see if that does something. (must be used to windows) [swright@localhost workspace]$ virtualenv botnet_etl New python executable in botnet_etl/bin/python Installing setuptoolsdone. Installing pip...done. [swright@localhost workspace]$ cd botnet_etl/ [swright@localhost botnet_etl]$ source bin/activate (botnet_etl)[swright@localhost botnet_etl]$ (botnet_etl)[swright@localhost botnet_etl]$ python -m site (botnet_etl)[swright@localhost botnet_etl]$ (botnet_etl)[swright@localhost botnet_etl]$ pip install -r stable-req.txt --- lots of text follows this for compiling and installing Install M2Crypto manually: Installed /home/swright/workspace/botnet_etl/lib/python2.6/site-packages/M2Crypto-0.21.1-py2.6-linux-x86_64.egg Processing dependencies for M2Crypto==0.21.1 Finished processing dependencies for M2Crypto==0.21.1 Install PySVN manually: (botnet_etl)[swright@localhost M2Crypto]$ cp -r ~/.virtualenvs/myvirtualenv/lib/python2.6/site-packages/pysvn ~/workspace/botnet_etl/lib/python2.6/site-packages/. (botnet_etl)[swright@localhost botnet_etl]$ python -m site (botnet_etl)[swright@localhost botnet_etl]$ -- http://mail.python.org/mailman/listinfo/python-list
Re: Virtualenv loses context
Thanks for the information Hans, I will double check that stuff. I am positive however that the environment was active. In addition that fact that I get different responses from python -m site from the botnet_etl virtual environment before I blew it away and after I rebuilt it really concerns me. -- http://mail.python.org/mailman/listinfo/python-list
Re: Virtualenv loses context
Brought my laptop out of hibernation to do some work this morning. I attempted to run one of my ETLs and got the following error. I made no changes since it was running yesterday. [swright@localhost app]$ python etl_botnet_meta.py --mode dev -f Traceback (most recent call last): File "etl_botnet_meta.py", line 9, in from make_zip import EtlMakeZip File "/home/swright/workspace/botnet_etl/app/make_zip.py", line 1, in import M2Crypto, os, time, datetime, json, hashlib, base64, zipfile ImportError: No module named M2Crypto [swright@localhost app]$ python -m site sys.path = [ '/home/swright/workspace/botnet_etl/app', '/usr/lib/python2.6/site-packages/pymongo-2.3-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/pip-1.2.1-py2.6.egg', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib/python2.6/site-packages', '/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info', ] USER_BASE: '/home/swright/.local' (exists) USER_SITE: '/home/swright/.local/lib/python2.6/site-packages' (doesn't exist) ENABLE_USER_SITE: True [swright@localhost app]$ -- http://mail.python.org/mailman/listinfo/python-list
