Re: [Tutor] Moving a conda environment to an off-line computer
Hi, The computer is a cluster with 4 nodes each running Linux (our Fedora-based distro), 12 cores/64 GB RAM per node using CUDA. At first, I'll be using only one node. -- Henrique C. S. Junior From: Tutor on behalf of Alan Gauld via Tutor Sent: Friday, November 30, 2018 22:33 To: tutor@python.org Subject: Re: [Tutor] Moving a conda environment to an off-line computer On 30/11/2018 13:47, Henrique Castro wrote: > ... use one of the powerful computers on my university It would help if we knew what kind of powerful computer this is. What OS does it run? It sounds like some variety of Unix, but which? Does it have a C compiler on board? > The computer does not have an internet connection > and I need to find a way to install a conda environment on it. This may be a question that the Anaconda community can better answer since building your own Anaconda suite is non trivial. So unless you find a pre-built environment for your new OS which you can download and copy to removable media you will likely need specialist help. That's more likely to be found on the Anaconda fora than on a general Python list like this one. > File > "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/http/client.py", > line 936, in connect > (self.host,self.port), self.timeout, self.source_address) > File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/socket.py", > line 704, in create_connection > for res in getaddrinfo(host, port, 0, SOCK_STREAM): > File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/socket.py", > line 745, in getaddrinfo > for res in _socket.getaddrinfo(host, port, family, type, proto, flags): > socket.gaierror: [Errno -2] Name or service not known Apparently the test file tries to access the internet so will obviously fail on a non internet connected device. (BTW is it on any kind of network or is it entirely standalone?) -- Alan G Author of the Learn to Program web site https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.alan-g.me.uk%2F&data=02%7C01%7C%7C2a739252872d48babd6208d65725104e%7C84df9e7fe9f640afb435%7C1%7C0%7C636792214023931869&sdata=8KuHyIuihRIwDB1bKIF7ZS7PJRwdtNLMVk7AYNOW%2B1s%3D&reserved=0 https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.amazon.com%2Fauthor%2Falan_gauld&data=02%7C01%7C%7C2a739252872d48babd6208d65725104e%7C84df9e7fe9f640afb435%7C1%7C0%7C636792214023931869&sdata=mgXxj1AChik%2B%2F2lCfL%2BNdJdfhUFDc2XLmEwgIMs8n50%3D&reserved=0 Follow my photo-blog on Flickr at: https://eur04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.flickr.com%2Fphotos%2Falangauldphotos&data=02%7C01%7C%7C2a739252872d48babd6208d65725104e%7C84df9e7fe9f640afb435%7C1%7C0%7C636792214023931869&sdata=HtkISkcgLNnKMFTIZcqT3VEZaBbPVfdRV11M16YjpQI%3D&reserved=0 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://eur04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Ftutor&data=02%7C01%7C%7C2a739252872d48babd6208d65725104e%7C84df9e7fe9f640afb435%7C1%7C0%7C636792214023931869&sdata=vrmETDxK13BxuOWQ2cSMG13xmb%2FX3ly85NvVwqEUjXg%3D&reserved=0 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Moving a conda environment to an off-line computer
On 01/12/2018 00:43, Henrique Castro wrote: >a cluster with 4 nodes each running Linux (our Fedora-based distro), OK, The good news is you should be able to get a pre-built distro rather than try to build your own. But I still think the Conda users are more likely to be able to assist. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Moving a conda environment to an off-line computer
Thank you guys, I'll try to contact the Conda community. Alan Gauld, good to know. I suggest that you try to spread the news to Fermi Lab, CERN and other research centers. Sometimes when you need a specific setting it is just easier to pre-build your distro than to repeat the same configuration hundreds of times. -- Henrique C. S. Junior From: Tutor on behalf of Alan Gauld via Tutor Sent: Saturday, December 1, 2018 07:12 To: tutor@python.org Subject: Re: [Tutor] Moving a conda environment to an off-line computer On 01/12/2018 00:43, Henrique Castro wrote: >a cluster with 4 nodes each running Linux (our Fedora-based distro), OK, The good news is you should be able to get a pre-built distro rather than try to build your own. But I still think the Conda users are more likely to be able to assist. -- Alan G Author of the Learn to Program web site https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.alan-g.me.uk%2F&data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435%7C1%7C0%7C636792525248838704&sdata=0xTl3WSOMgfGih2FYN8AYSzXORCRy9fKLN6iLk3txlk%3D&reserved=0 https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.amazon.com%2Fauthor%2Falan_gauld&data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435%7C1%7C0%7C636792525248838704&sdata=FVJGUemjFp3fRyu%2FK7PpkefITCLyjLmZ1DGuhmfLqtU%3D&reserved=0 Follow my photo-blog on Flickr at: https://nam03.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.flickr.com%2Fphotos%2Falangauldphotos&data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435%7C1%7C0%7C636792525248838704&sdata=UTIgec8%2BlrRQ75j8j%2FFla5CYsU7%2FWywZb0D4kY6Gefs%3D&reserved=0 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Ftutor&data=02%7C01%7C%7C29a2850e262c4fd9148c08d6576d86d0%7C84df9e7fe9f640afb435%7C1%7C0%7C636792525248838704&sdata=lR6inUjJWRGm9TZ2MwyoWXLkXSinM1lCbkCcJV6MGuw%3D&reserved=0 ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Moving a conda environment to an off-line computer
> On Dec 1, 2018, at 05:41, Henrique Castro wrote: > > Thank you guys, I'll try to contact the Conda community. > Alan Gauld, good to know. I suggest that you try to spread the news to Fermi > Lab, CERN and other research centers. > Sometimes when you need a specific setting it is just easier to pre-build > your distro than to repeat the same configuration hundreds of times. So it sounds like you have local network connectivity, but do not have external internet access? I ask, because I wonder how you are copying the data to the system from your computer that does have an internet connection. Is there no internet access at all? By that, I mean is there also no web proxy that can be used? If a proxy exists, it may be possible to do something like define an http_proxy os.environ['http_proxy’]=‘yourproxy:port' It looks to me like you have a bigger problem than being able to “install conda.” Even after conda is functioning, errors like : Traceback (most recent call last): File "qm7_ANI.py", line 15, in featurizer='BPSymmetryFunction', split='stratified', move_mean=False) File "/home/henrique/bin/anaconda3/envs/deepchem/lib/python3.6/site-packages/deepchem/molnet/load_function/qm7_datasets.py", line 50, in load_qm7_from_mat 'http://deepchem.io.s3-website-us-west-1.amazonaws.com/datasets/qm7.mat' tell me it is always going to want to use the internet to access datasets in AWS. Unless you can use a web proxy, or possibly get a local mirror of the deepchem.io data on a university system you _can_ reach, this may not be possible to resolve. — David Rock da...@graniteweb.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Moving a conda environment to an off-line computer
On 01/12/2018 11:41, Henrique Castro wrote: > Thank you guys, I'll try to contact the Conda community. > Alan Gauld, good to know. I suggest that you try to spread the news I only meant that because it was Fedora (rather than a bespoke Linux build) that you should find a binary package someplace. You still need to figure out how to install it on your cluster, but that's still much easier than building everything from scratch. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Moving a conda environment to an off-line computer
On 12/1/18 12:20 PM, Alan Gauld via Tutor wrote: > On 01/12/2018 11:41, Henrique Castro wrote: >> Thank you guys, I'll try to contact the Conda community. > > >> Alan Gauld, good to know. I suggest that you try to spread the news > > I only meant that because it was Fedora (rather than a bespoke > Linux build) that you should find a binary package someplace. > You still need to figure out how to install it on your cluster, > but that's still much easier than building everything from scratch. > > yes, fedora has an anaconda package. but since anaconda itself is a kind of package manager, the interesting thing is getting it to install the stuff you want, and that stuff is not going to come from Fedora packages, even if there may in fact be packages of the things you want to install - the fedpkgs wouldn't put stuff in the places anaconda will expect. conda caches packages it has downloaded. I don't know exactly where, but I think it will tell you if you ask it to dump its full configuration. if you replicate the cache directory from a test machine which is internet-connected over to the cluster, it seems likely installs from inside conda will then work. hopefully :) ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Moving a conda environment to an off-line computer
On Sat, 1 Dec 2018 at 00:10, Henrique Castro wrote: > > Dear colleagues, > Soon I'll start to use one of the powerful computers on my university as a > tool in my Ph.D. The computer does not have an internet connection and I need > to find a way to install a conda environment on it. When you say that you need to install a conda environment I imagine that what you mean is you need to install some Python packages and you think the easiest way is to install a conda environment. Is that correct? Having used a similar setup at my University the situation we have is that there is a big cluster with thousands of nodes that don't have internet access but there is internet access on certain nodes called the "login nodes" which you can ssh into. They have a filesystem that is shared with all other nodes which means you can use git, pip etc to get your code set up and working before submitting a job to be run on the main cluster. Is that not the case for the setup you're using? Also the setup we have actually provides many Python versions and packages: I just have to activate them with a command that's something like "module activate python-3.6 python-3.6-numpy ...". The other aspect of our setup is that it is well equipped with compilers, scientific libraries etc. so that it's usually straight forward to compile e.g. numpy from source with "python setup.py install". I don't know if I could also just transfer manylinux wheels in there to avoid compiling as well... So there are potentially a number of ways of achieving what you want. With the information you've provided so far, I'm not clear what the best way forward is. The first question is why is it that you want a conda environment? -- Oscar ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] Moving a conda environment to an off-line computer
On 01/12/2018 11:41, Henrique Castro wrote: > Thank you guys, I'll try to contact the Conda community. Try: https://support.anaconda.com/ "Community Support" looks a likely option -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.amazon.com/author/alan_gauld Follow my photo-blog on Flickr at: http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
