Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Dag Sverre Seljebotn
On 01/14/2012 12:28 AM, Sturla Molden wrote: > Den 13.01.2012 22:42, skrev Sturla Molden: >> Den 13.01.2012 22:24, skrev Robert Kern: >>> Do these systems have a ramdisk capability? >> I assume you have seen this as well :) >> >> http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Sturla Molden
Den 13.01.2012 22:42, skrev Sturla Molden: > Den 13.01.2012 22:24, skrev Robert Kern: >> Do these systems have a ramdisk capability? > I assume you have seen this as well :) > > http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_final.pdf > This paper also repeats a common mistak

Re: [Numpy-discussion] [JOB] Extracting subset of dataset using latitude and longitude

2012-01-13 Thread Chao YUE
Hi, I don't know if numpy has ready tool for this. I also have this use in my study. So I write a simple code for my personal use. It might no be great. I hope others can also respond as this is very basic function in earth data analysis. ##

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Robert Kern
On Fri, Jan 13, 2012 at 21:42, Sturla Molden wrote: > Den 13.01.2012 22:24, skrev Robert Kern: >> Do these systems have a ramdisk capability? > > I assume you have seen this as well :) > > http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_final.pdf I hadn't, actually! Good find

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Langton, Asher
On 1/13/12 1:58 PM, Dag Sverre Seljebotn wrote: > >It's actually not too difficult to do something like > >LD_PRELOAD=myhack.so python something.py > >and have myhack.so intercept the filesystem calls Python makes (to libc) >and do whatever it wants. That's a solution that doesn't interfer with >ho

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Dag Sverre Seljebotn
On 01/13/2012 10:20 PM, Langton, Asher wrote: > On 1/13/12 12:38 PM, Sturla Molden wrote: >> Den 13.01.2012 21:21, skrev Dag Sverre Seljebotn: >>> Another idea: Given your diagnostics, wouldn't dumping the output of >>> "find" of every path in sys.path to a single text file work well? >> >> It prob

[Numpy-discussion] [JOB] Extracting subset of dataset using latitude and longitude

2012-01-13 Thread Jeremy Lounds
Hello, I am looking for some help extracting a subset of data from a large dataset. The data is being read from a wgrib2 (World Meterological Organization standard gridded data) using the pygrib library. The data values, latitudes and longitudes are in separate lists (arrays?), and I would lik

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Langton, Asher
On 1/13/12 1:24 PM, Robert Kern wrote: >On Fri, Jan 13, 2012 at 21:20, Langton, Asher wrote: > >> 2) More generally, dealing with this as well as other library-loading >> issues at the system level, perhaps by putting a small disk near a node >>or >> small collection of nodes, along with a command

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Travis Oliphant
It is a straightforward thing to implement a "registry mechanism" for Python that by-passes imp.find_module (i.e. using sys.meta_path). You could imagine creating the registry file for a package or distribution (much like Dag described) and push that to every node during distribution. The

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Sturla Molden
Den 13.01.2012 22:24, skrev Robert Kern: > Do these systems have a ramdisk capability? I assume you have seen this as well :) http://www.cs.uoregon.edu/Research/paracomp/papers/iccs11/iccs_paper_final.pdf Sturla ___ NumPy-Discussion mailing list NumP

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Robert Kern
On Fri, Jan 13, 2012 at 21:20, Langton, Asher wrote: > 2) More generally, dealing with this as well as other library-loading > issues at the system level, perhaps by putting a small disk near a node or > small collection of nodes, along with a command to push (broadcast) some > portions of the fi

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Langton, Asher
On 1/13/12 12:38 PM, Sturla Molden wrote: >Den 13.01.2012 21:21, skrev Dag Sverre Seljebotn: >> Another idea: Given your diagnostics, wouldn't dumping the output of >> "find" of every path in sys.path to a single text file work well? > >It probably would, and would also be less prone to synchroniza

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Sturla Molden
Den 13.01.2012 21:21, skrev Dag Sverre Seljebotn: > Another idea: Given your diagnostics, wouldn't dumping the output of > "find" of every path in sys.path to a single text file work well? It probably would, and would also be less prone to synchronization problems than using an MPI broadcast. Ano

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Dag Sverre Seljebotn
On 01/13/2012 09:19 PM, Dag Sverre Seljebotn wrote: > On 01/13/2012 02:13 AM, Asher Langton wrote: >> Hi all, >> >> (I originally posted this to the BayPIGgies list, where Fernando Perez >> suggested I send it to the NumPy list as well. My apologies if you're >> receiving this email twice.) >> >> I

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Dag Sverre Seljebotn
On 01/13/2012 02:13 AM, Asher Langton wrote: > Hi all, > > (I originally posted this to the BayPIGgies list, where Fernando Perez > suggested I send it to the NumPy list as well. My apologies if you're > receiving this email twice.) > > I work on a Python/C++ scientific code that runs as a number o

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Robert Kern
On Fri, Jan 13, 2012 at 19:41, Sturla Molden wrote: > Den 13.01.2012 02:13, skrev Asher Langton: >> intern try a few of these approaches. It turned out that the problem >> wasn't so much the simultaneous module loads, but rather the huge >> number of failed open() calls (ENOENT) as the interpreter

Re: [Numpy-discussion] Improving Python+MPI import performance

2012-01-13 Thread Sturla Molden
Den 13.01.2012 02:13, skrev Asher Langton: > intern try a few of these approaches. It turned out that the problem > wasn't so much the simultaneous module loads, but rather the huge > number of failed open() calls (ENOENT) as the interpreter tries to > find the module files. It sounds like there i

Re: [Numpy-discussion] Python for Scientists - courses in Germany and US

2012-01-13 Thread Ilan Schnell
By the way, Enthought is also offering Python training and we have just updated out training calendar for this year: http://www.enthought.com/training/enthought_training_calendar.php We are offering about 20 open Python classes in the US and Europe this year. - Ilan On Fri, Jan 13, 2012 at 4:32

[Numpy-discussion] Python for Scientists - courses in Germany and US

2012-01-13 Thread Mike Müller
Learn NumPy and Much More = Scientists like Python. If you would like to learn more about important libraries for scientific applications, you might be interested in these courses. The course in Germany covers: - Overview of libraries - NumPy - Data storage with text file

Re: [Numpy-discussion] Question on F/C-ordering in numpy svd

2012-01-13 Thread Pearu Peterson
On 01/12/2012 04:21 PM, Ivan Oseledets wrote: > Dear all! > > I quite new to numpy and python. > I am a matlab user, my work is mainly > on multidimensional arrays, and I have a question on the svd function > from numpy.linalg > > It seems that > > u,s,v=svd(a,full_matrices=False) > > returns u a