Re: [Tutor] Set LD_LIBRARY_PATH and equivalents platform-independently

[Alan Gauld]

On 17/01/13 02:10, Dave Angel wrote:


I don't recall enough about Windows to be sure whether putenv would
work, but the environment variable Windows uses to search for DLL's is
certainly not LD_LIBRARY_PATH


If you check the code Albert is actually using different variables per 
platform. For Windows he is using PATH...



--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] learning to program in cython

[Albert-Jan Roskam]

>> With the help of an awesome python community I have been able to pick up the

>> language and now willing to explore other cool extensions of it.
>
>Good work!
>
>>
>> I routinely have large loops which could be ported to cython for speed.
>> However I have never written a single line of cython code. Any pointers on
>> getting started.
>
>There are two reasons for using cython:
>1) To interface directly with existing C libraries.
>2) To speed up CPU-intensive code.
>
>It sounds like you're interested in case 2). However, not all loops
>benefit from cythonisation. Loosely cython is good when
>a) you're doing something that can be written in a small amount of
>efficient C code
>b) but the corresponding Python code involves a lot of repeated
>function calls or expression evaluations.
>
>If you're already familiar with C then you'll probably have some idea
>when a) and b) apply. I would say that a prerequisite for learning to
>speed up CPU-intensive code with cython would be learning to use the
>python profilers. In particular you should learn to use cProfile and
>timeit:

(and of course pstats)

I recently used Cython for the first time and I found it surprisingly easy. The 
installation under Linux
is easy, but seems to be not really trivial under Windows (I never tried 
installing it under windows though; I'd first try one of the unofficial 
binaries: http://www.lfd.uci.edu/~gohlke/pythonlibs/#cython). One gotcha that I 
only found out later: it is possible to generate an html report of your 
Cythonized code. The yellower the code is coloured, the more it might still 
benefit from speed improvements: 
http://stackoverflow.com/questions/11058933/cython-a-flag-to-generate-yellow-shaded-html-without-command-line

It's pretty cool to use cProfile to make a Cython-Python comparison of the 
function you're trying to speed up. In my case it was 87 % faster IIRC (this 
difference was not only significant, but also relevent: a use case where this 
would save 15 minutes is totally realistic) . Yaay!

Albert-Jan
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Set LD_LIBRARY_PATH and equivalents platform-independently

[Albert-Jan Roskam]

> Subject: Re: [Tutor] Set LD_LIBRARY_PATH and equivalents 
> platform-independently

> 
> On 01/16/2013 07:05 PM, Alan Gauld wrote:
>>  On 16/01/13 21:06, Albert-Jan Roskam wrote:
>> 
>>>  Is there a builtin function that can set LD_LIBRARY_PATH or equivalents
>>>  platform-independently?
>> 
>>  No.
>>  But there is putenv() in the os module.
>>  But how well it works across OS I'm not sure.
>> 
> 
> I don't recall enough about Windows to be sure whether putenv would 
> work, but the environment variable Windows uses to search for DLL's is 
> certainly not LD_LIBRARY_PATH

Hi Alan, Dave,

Thanks for your replies. os.putenv() may be easier than os.environ because, 
hopefully, it takes care of the OS-specific separators of the values (";" for 
Windows, ":" for Linux, others I don't know but I'd guess they're all ":").  
Then again, this sentence from the Python page is a little worrying: "*If* the 
platform supports the putenv() function, ...". (emphasis added). But os.environ 
has its own problems: "On some platforms, including FreeBSD and Mac OS X, 
setting environ may
cause memory leaks.  Refer to the system documentation for putenv()." H.

As an alternative, I used os.chdir to tell the OS where to start looking for 
libraries, but somebody on this list (I forgot who) warned me that this could 
have nasty side effects.

http://docs.python.org/2/library/os.html

os.environ
A mapping object representing the string environment. For example, 
environ['HOME'] is the pathname of your home directory (on some platforms),
and is equivalent to getenv("HOME") in C.
This mapping is captured the first time the os module is imported,
typically during Python startup as part of processing site.py.  Changes
to the environment made after this time are not reflected in os.environ,
except for changes made by modifying os.environ directly.


If the platform supports the putenv() function, this mapping may be used
to modify the environment as well as query the environment. putenv() will
be called automatically when the mapping is modified.


Note. Calling putenv() directly does not change os.environ, so it’s better
to modify os.environ.

Note. On some platforms, including FreeBSD and Mac OS X, setting environ may
cause memory leaks.  Refer to the system documentation for putenv().
If putenv() is not provided, a modified copy of this mapping  may be
passed to the appropriate process-creation functions to cause  child processes
to use a modified environment.

Regards,
Albert-Jan
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Set LD_LIBRARY_PATH and equivalents platform-independently

[eryksun]

On Wed, Jan 16, 2013 at 4:06 PM, Albert-Jan Roskam  wrote:
>
> Is there a builtin function that can set LD_LIBRARY_PATH or equivalents
> platform-independently? It would be nice use such a function in a setup
> script.

Modifying LD_LIBRARY_PATH only affects child processes (ld.so caches
the search path). In contrast, Windows evaluates the current PATH when
searching for DLLs (other than SxS assemblies). What's the immediate
goal here? For building an extension module with distutils you can use
library_dirs and runtime_library_dirs (UNIX, embed an rpath).
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Set LD_LIBRARY_PATH and equivalents platform-independently

[eryksun]

On Thu, Jan 17, 2013 at 7:14 AM, Albert-Jan Roskam  wrote:
>
> Thanks for your replies. os.putenv() may be easier than os.environ because,
> hopefully, it takes care of the OS-specific separators of the values (";"
> for Windows, ":" for Linux, others I don't know but I'd guess they're all
> ":").

os.environ wraps the initial posix.environ dict and uses
putenv/unsetenv to keep it consistent with the process:

http://hg.python.org/cpython/file/70274d53c1dd/Lib/os.py#l391

convertenviron in posixmodule.c (module posix, nt, or os2) populates
the initial dict:

http://hg.python.org/cpython/file/70274d53c1dd/Modules/posixmodule.c#l442

For the path separator, use os.pathsep or os.path.pathsep:

http://docs.python.org/2/library/os#os.pathsep
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Set LD_LIBRARY_PATH and equivalents platform-independently

[Albert-Jan Roskam]

 Original Message -

> From: eryksun 
> To: Albert-Jan Roskam 
> Cc: Python Mailing List 
> Sent: Thursday, January 17, 2013 3:21 PM
> Subject: Re: [Tutor] Set LD_LIBRARY_PATH and equivalents 
> platform-independently
> 
> On Wed, Jan 16, 2013 at 4:06 PM, Albert-Jan Roskam  
> wrote:
>> 
>>  Is there a builtin function that can set LD_LIBRARY_PATH or equivalents
>>  platform-independently? It would be nice use such a function in a setup
>>  script.
> 
> Modifying LD_LIBRARY_PATH only affects child processes (ld.so caches
> the search path). In contrast, Windows evaluates the current PATH when
> searching for DLLs (other than SxS assemblies). What's the immediate
> goal here? For building an extension module with distutils you can use
> library_dirs and runtime_library_dirs (UNIX, embed an rpath)

Hi Eryksun,

The goal is to load the C libraries (dll, so, dylib, etc) that my program 
needs. Your suggestion about distutils seems very useful! I've also been 
checking distutils2, setuptools and distribute and I still wasn't sure which 
package to choose (distutils2.core doesn't seem to exist anymore, so I'll just 
try to use distutils).

Anyway, I looked up your two suggestions about library_dirs and 
runtime_library_dirs. What is meant by "at link time"? (it might be that I 
don't understand this because English is not my mother tongue). Given your 
explanation, the "runtime_library_dirs" does not seem to be an option.

Then your remark about rpath. Does this mean that the ELF header of the library 
itself is modified (I believe I read something about that on StackOverflow)? 
The libraries I am using are copyrighted (one can freely use them, but no 
reverse engineering, disentangling, etc). I am not sure whether adding an rpath 
will be copyright infringement. Logically, I'd say no, but I doubt whether 
logic applies in legal stuff. ;-)

   library_dirs : [string]
 |  list of directories to search for C/C++ libraries at link time
 |    libraries : [string]
 |  list of library names (not filenames or paths) to link against
 |    runtime_library_dirs : [string]
 |  list of directories to search for C/C++ libraries at run time
 |  (for shared extensions, this is when the extension is loaded)

Regards,
Albert-Jan
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Set LD_LIBRARY_PATH and equivalents platform-independently

[Alan Gauld]

On 17/01/13 12:14, Albert-Jan Roskam wrote:


Thanks for your replies. os.putenv() may be easier than os.environ because,

> hopefully, it takes care of the OS-specific separators of the values
> (";" for Windows, ":" for Linux, others I don't know

I wouldn't count on it. Support for changing environment variables on 
the fly is iffy in most languages and is not something I'd ever do lightly.


The environment is something that usually should be set up when the user 
logs in to reflect their preferences, its not really polite of a program 
to try and alter the users environment...



Then again, this sentence from the Python page is a little worrying:

> "*If* the platform supports the putenv() function, ...".

Not all OS allow changes to the environment variables. Others only allow 
the value to be a fixed size allocated at startup and if you write more 
bytes than are already assigned you will overwrite the next value! The 
OS assumes that this stuff is set up once and not changed.


putenv() should always be considered a platform specific bit of code.



As an alternative, I used os.chdir to tell the OS where to start

> looking for libraries, but somebody on this list (I forgot who)
> warned me that this could have nasty side effects.

Again the user may expect to be in a certain directory so as a minimum 
you need to change back to where they were when you exit.


The normal way to deal with local file locations is to have a resource 
file that's read when the program starts up - like .vimrc or .emacs
Then if the file doesn't exist you either create it at install time or 
use default values and allow the user to override them.


HTH

--
Alan G
Author of the Learn to Program web site
http://www.alan-g.me.uk/

___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] learning to program in cython

[Oscar Benjamin]

On 17 January 2013 11:52, Albert-Jan Roskam  wrote:
>
> I recently used Cython for the first time and I found it surprisingly easy. 
> The installation under Linux
> is easy, but seems to be not really trivial under Windows (I never tried 
> installing it under windows though; I'd first try one of the unofficial 
> binaries: http://www.lfd.uci.edu/~gohlke/pythonlibs/#cython).

I've never had a problem installing cython under Windows. Although,
cython does require a Python-compatible C compiler. This can be (the
right version of) MSVC or (with a little fiddling) mingw32. I use
mingw32 and, although it is not officially supported by CPython I
think it is officially supported by Cython. I have had problems
getting the right C-compiler on Windows.

In any case, once you have a C compiler installed it should be
straightforward to install cython:
1) Get the archive from here: http://cython.org/#download
2) Extract it
3) python setup.py install

If you have pip or easy_install then it's as easy as running 'pip
install cython' or 'easy_install cython' in your terminal.


Oscar
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Json encode and decode on Puython 2.4

[Dotan Cohen]

On Thu, Jan 17, 2013 at 1:15 AM, Steven D'Aprano  wrote:
> Python 2.4 is no longer receiving security updates. If you're exposing a
> web app on the Internet using Python 2.4, it's just a matter of time
> before you're hacked.
>
> Time to change hosting companies, methinks.
>

The time to get rid of this host has long passed, alas it is not my
decision! In any case the only thing running on it are non-critical
stuff used to test that the critical stuff hosted on a real server are
working as intended. Nothing facing the bid bad web.

Thanks.

-- 
Dotan Cohen

http://gibberish.co.il
http://what-is-what.com
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

[Tutor] Warning for users of the Python and Jython wiki

[Steven D'Aprano]

Hello all,

Some time recently, the wiki at http://wiki.python.org/ was hacked. The
vandal who broke in deleted all the wiki data. However, it is possible
that before destroying the data, he may have gained access to user
passwords.

If you had an account on the wiki, and use the same password elsewhere,
then you should change that password immediately.

In general, you should not use the same password for multiple systems,
and naturally once a password has been (potentially) compromised, never
use it, or trivial modifications of it, again.

http://thehackernews.com/2013/01/official-debian-and-python-wiki-servers.html#_



--
Steven
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] Set LD_LIBRARY_PATH and equivalents platform-independently

[eryksun]

On Thu, Jan 17, 2013 at 10:33 AM, Albert-Jan Roskam  wrote:
>
> The goal is to load the C libraries (dll, so, dylib, etc) that my program
> needs.
>
> Anyway, I looked up your two suggestions about library_dirs and
> runtime_library_dirs. What is meant by "at link time"?

library_dirs adds search paths for the linker (e.g. ld) for finding
shared libraries (.so) and static libraries (.a archives of
relocatable object files).

Linking to a shared library is handled by the runtime loader/linker
(e.g. /lib/ld-linux.so.2). For Linux, ld.so searches the
system-defined directories (/lib, /usr/lib, and those set by
/etc/ld.so.conf), which are cached in /etc/ld.so.cache (created by
ldconfig). This is where distutils runtime_library_dirs comes into
play. For ELF it configures an embedded RPATH, which the loader
searches before the system directories. You can also convert the RPATH
to a RUNPATH, which can be overridden by LD_LIBRARY_PATH.

> Does this mean that the ELF header of the library itself is modified

readelf -d shows the .dynamic section (including strings from
.dynstr), which you can use to verify the RPATH/RUNPATH. chrpath lets
you change (but not add) an RPATH, up to its existing length. It also
lets you convert an RPATH to a RUNPATH. patchELF can add or extend an
RPATH/RUNPATH.

http://nixos.org/patchelf.html

> The libraries I am using are copyrighted (one can freely use them, but
> no reverse engineering, disentangling, etc). I am not sure whether
> adding an rpath will be copyright infringement. Logically, I'd say no,
> but I doubt whether logic applies in legal stuff.

It should be OK for internal administration. You'd have to ask a legal
expert about distribution.

If you stick with LD_LIBRARY_PATH, etc, keep in mind it has to be set
before the process starts, typically in a wrapper script. On Windows,
however, you can modify PATH at runtime.
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor

Re: [Tutor] HELP- Regarding working with python

[Gayathri S]

hi...
  I am using principal component analysis for dimensionality
reduction in python. am having this following error...
>>> import numpy as np
>>> import matplotlib.pyplot as plt
>>> import mlpy
>>> np.random.seed(0)
>>> mean,cov,n=[0,0],[[1,1],[1,1.5]],100
>>> x=np.random.multivariate_normal(mean,cov,n)
>>> pca.learn(x)
Traceback (most recent call last):
  File "", line 1, in 
NameError: name 'pca' is not defined
>>>
would you please help me in finding the error...? what was my fault? how
could i use PCA in python..?

  Thanks.!

On Wed, Jan 9, 2013 at 12:58 PM, Gayathri S  wrote:

> Hi..
> I would like to use Principal component analysis independent
> component analysis in python. Wanna know whether it will support this
> efficiently or not?
>
>
> On Wed, Jan 2, 2013 at 4:59 PM, Alan Gauld wrote:
>
>> On 02/01/13 07:20, Gayathri S wrote:
>>
>>> Hi..
>>>   I am using python 2.7 and scikit-learn for machine learning.
>>> And OS is Windows 7. Wanna know how to import our own data sets  in
>>> scikit-learn?
>>>
>>
>> Further to my last mail there is a gmane group
>>
>> gmane.comp.python.scikit-learn
>>
>> I'd try looking there, or wherever it is sourced originally.
>>
>>
>>
>> --
>> Alan G
>> Author of the Learn to Program web site
>> http://www.alan-g.me.uk/
>>
>> __**_
>> Tutor maillist  -  Tutor@python.org
>> To unsubscribe or change subscription options:
>> http://mail.python.org/**mailman/listinfo/tutor
>>
>
>
>
> --
>
>
>
>
> Keep Smiling.
> Regards
> Gayu
>



-- 




Keep Smiling.
Regards
Gayu
___
Tutor maillist  -  Tutor@python.org
To unsubscribe or change subscription options:
http://mail.python.org/mailman/listinfo/tutor