Re: [Numpy-discussion] python geospatial package?
2012/2/23 Vincent Schut > On 02/22/2012 10:45 PM, Chao YUE wrote: > > Hi all, > > > > Is anyone using some python geospatial package that can do jobs like > > intersection, etc. the job is like you automatically extract a region > > on a global map etc. > > > > thanks and cheers, > > > > Chao > > Depending what you want to do: Shapely, GDAL/OGR, pyproj, Mapnik, Basemap,... ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Reading a big netcdf file
Hi. I'm trying to read a big netcdf file (445 Mb) using netcdf4-python. The data are described as: *The GEBCO gridded data set is stored in NetCDF as a one dimensional array of 2-byte signed integers that represent integer elevations in metres. The complete data set gives global coverage. It consists of 21601 x 10801 data values, one for each one minute of latitude and longitude for 233312401 points. The data start at position 90°N, 180°W and are arranged in bands of 360 degrees x 60 points/degree + 1 = 21601 values. The data range eastward from 180°W longitude to 180°E longitude, i.e. the 180° value is repeated.* The problem is that it is very slow (or I am quite newbie). Anyone has a suggestion to get these data in a numpy array in a faster way? Thanks in advance. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Reading a big netcdf file
Hi, all.
Thank you very much for your replies.
I am obtaining some issues. If I use netcdf4-python or scipy.io.netcdf
libraries:
In [4]: import netCDF4 as n4
In [5]: from scipy.io import netcdf as nS
In [6]: import numpy as np
In [7]: gebco4 = n4.Dataset('GridOne.grd', 'r')
In [8]: gebcoS = nS.netcdf_file('GridOne.grd', 'r')
Now, if a do:
In [9]: z4 = gebco4.variables['z']
I got no problems and I have:
In [14]: type(z4); z4.shape; z4.size
Out[14]:
Out[14]: (233312401,)
Out[14]: 233312401
But if I do:
In [15]: z4 = gebco4.variables['z'][:]
Traceback (most recent call last):
File "", line 1, in
File "netCDF4.pyx", line 2466, in netCDF4.Variable.__getitem__
(netCDF4.c:22943)
File "C:\Python26\lib\site-packages\netCDF4_utils.py", line 278, in
_StartCountStride
n = len(range(beg,end,inc))
MemoryError
I got a memory error. But if a select a smaller array I've got:
In [16]: z4 = gebco4.variables['z'][:1000]
In [17]: type(z4); z4.shape; z4.size
Out[17]:
Out[17]: (1000,)
Out[17]: 1000
What's the difference between z4 as a netCDF4.Variable and as a
numpy.ndarray?
Now, if I use scipy.io.netcdf:
In [18]: zS = gebcoS.variables['z']
In [20]: type(zS); zS.shape
Out[20]:
Out[20]: (233312401,)
In [21]: zS = gebcoS.variables['z'][:]
In [22]: type(zS); zS.shape
Out[22]:
Out[22]: (233312401,)
What's the difference between zS as a scipy.io.netcdf.netcdf_variable and as
a numpy.ndarray?
Why with scipy.io.netcdf I do not have a MemoryError?
Finally, if I do the following (maybe it's a silly thing do this) using Eric
suggestions to clear the cache:
In [32]: zS = gebcoS.variables['z']
In [38]: timeit -n1 -r1 zSS = np.array(zS[:1]) # 100.000.000 out of
233.312.401 because I've got a MemoryError
1 loops, best of 1: 73.1 s per loop
(If I use a copy, timeit -n1 -r1 zSS = np.array(zS[:1], copy=True),
I get a MemoryError and I have to set the size to 50.000.000 but it's quite
fast).
Than you very much for your replies and excuse me if some questions are very
basic.
Best regards.
***
The results of ncdump -h
netcdf GridOne {
dimensions:
side = 2 ;
xysize = 233312401 ;
variables:
double x_range(side) ;
x_range:units = "user_x_unit" ;
double y_range(side) ;
y_range:units = "user_y_unit" ;
short z_range(side) ;
z_range:units = "user_z_unit" ;
double spacing(side) ;
short dimension(side) ;
short z(xysize) ;
z:scale_factor = 1. ;
z:add_offset = 0. ;
z:node_offset = 0 ;
// global attributes:
:title = "GEBCO One Minute Grid" ;
:source = "1.02" ;
}
The file is publicly available from:
http://www.gebco.net/data_and_products/gridded_bathymetry_data/
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] what python module to modify NetCDF data?
Quoting Chao YUE : > > > Dear all, > > > > I want to change some variable values in a series of NetCDF file. Did > > anybody else did this before using python? > > Now I use pupynere for reading data from NetCDF files and making plots. > but > > the document of pupynere for writing data to NetCDF file is quite simple > and > > I still feel difficult to do this with pupynere. > > > > the NetCDF file I want to change is a global data (0.5X0.5d resolution, > > 360X720grid with 12 time steps) and have approx. 10 variables. I just > want > > to change some points for a specific > > variable for all 12 time steps. I know it's possible use NCO ncap2 > utility > > to do the job. but now I have some problem in using ncap2 within a shell > > script. > > I guess there is some easy way to use some python module to do the job? > like > > mainly altering the data that need to change while let the others > remaining > > intact? > > > > Any idea will be greatly appreciated. I will all a good weekend, > > > > Chao > Hi. Have a look to [1] and [2]. [1] http://code.google.com/p/netcdf4-python/ [2] http://www.scipy.org/doc/api_docs/SciPy.io.netcdf.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fwd: Re: Creating parallel curves
2012/2/13 Andrea Gavana > -- Forwarded message -- > From: "Andrea Gavana" > Date: Feb 13, 2012 11:31 PM > Subject: Re: [Numpy-discussion] Creating parallel curves > To: "Jonathan Hilmer" > > Thank you Jonathan for this, it's exactly what I was looking for. I' ll > try it tomorrow on the 768 well trajectories I have and I'll let you know > if I stumble upon any issue. > > If someone could shed some light on my problem number 2 (how to adjust the > scaling/distance) so that the curves look parallel on a matplotlib graph > even though the axes scales are different, I'd be more than grateful. > > Thank you in advance. > Hi. Maybe this could help you as a starting point. *from Shapely.geometry import LineString from matplotlib import pyplot myline = LineString(...) x, y = myline.xy xx, yy = myline.buffer(distancefrommyline).exterior.xy # coordinates around myline pyplot.plot(x, y) pyplot.plot(xx,yy) pyplot.show()* Best. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Calculation of a hessian
Hi all, I am trying to calculate a Hessian. I am using numdifftools for this ( https://pypi.python.org/pypi/Numdifftools). My question is, is it possible to make it using pure numpy?. The actual code is like this: *import numdifftools as nd* *import numpy as np* *def log_likelihood(params):* *sum1 = 0; sum2 = 0* *mu = params[0]; sigma = params[1]; xi = params[2]* *for z in data:* *x = 1 + xi * ((z-mu)/sigma)* *sum1 += np.log(x)* *sum2 += x**(-1.0/xi)* *return -((-len(data) * np.log(sigma)) - (1 + 1/xi)*sum1 - sum2) # negated so we can use 'minimum'* *kk = nd.Hessian(log_likelihood)* Thanks in advance. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Calculation of a hessian
2014-08-08 11:51 GMT+02:00 Jose Gomez-Dans : > Your function looks fairly simple to differentiate by hand, but if you > have access to the gradient (or you estimate it numerically using > scipy...), this function might do the job: > > def hessian ( x, the_func, epsilon=1e-8): > """Numerical approximation to the Hessian > Parameters > > x: array-like > The evaluation point > the_func: function > The function. We assume that the function returns the function > value and > the associated gradient as the second return element > epsilon: float > The size of the step > """ > > N = x.size > h = np.zeros((N,N)) > df_0 = the_func ( x )[1] > for i in xrange(N): > xx0 = 1.*x[i] > x[i] = xx0 + epsilon > df_1 = the_func ( x )[1] > h[i,:] = (df_1 - df_0)/epsilon > x[i] = xx0 > return h > > Jose > > Hi José, Thanks for the answer. My idea would be to generalise the calculation of the Hessian, not just to differentiate the example I posted and I was wondering if Numpy/Scipy already had something similar to that provided by NumDiffTools. Thanks again. > > On 8 August 2014 08:31, Kiko wrote: > >> Hi all, >> >> I am trying to calculate a Hessian. I am using numdifftools for this ( >> https://pypi.python.org/pypi/Numdifftools). >> >> My question is, is it possible to make it using pure numpy?. >> >> The actual code is like this: >> >> >> *import numdifftools as nd* >> *import numpy as np* >> >> *def log_likelihood(params):* >> *sum1 = 0; sum2 = 0* >> *mu = params[0]; sigma = params[1]; xi = params[2]* >> *for z in data:* >> *x = 1 + xi * ((z-mu)/sigma)* >> *sum1 += np.log(x)* >> *sum2 += x**(-1.0/xi)* >> *return -((-len(data) * np.log(sigma)) - (1 + 1/xi)*sum1 - sum2) # >> negated so we can use 'minimum'* >> >> *kk = nd.Hessian(log_likelihood)* >> >> Thanks in advance. >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Calculation of a hessian
2014-08-08 16:37 GMT+02:00 Eelco Hoogendoorn : > Do it in pure numpy? How about copying the source of numdifftools? > Of course it is a solution. I was just wondering if it exist something similar in the numpy/scipy packages so I do not have to use a new third party library to do that. > What exactly is the obstacle to using numdifftools? There seem to be no > licensing issues. In my experience, its a crafty piece of work; and > calculating a hessian correctly, accounting for all kinds of nasty floating > point issues, is no walk in the park. Even if an analytical derivative > isn't too big a pain in the ass to implement, there is a good chance that > what numdifftools does is more numerically stable (though in all likelihood > much slower). > > The only good reason for a specialized solution I can think of is speed; > but be aware what you are trading it in for. If speed is your major concern > though, you really cant go wrong with Theano. > > > http://deeplearning.net/software/theano/library/gradient.html#theano.gradient.hessian > > Thanks, it seems that NumDiffTools is the way to go. > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] netcdf lat lon to coord - ValueError: need more than 1 value to unpack
2015-03-24 11:02 GMT+01:00 questions anon : > I would like to find the nearest coord in a netcdf from a given latitude > and longitude. > I found some fantastic code that does this - > http://nbviewer.ipython.org/github/Unidata/unidata-python-workshop/blob/master/netcdf-by-coordinates.ipynb > but I keep receiving this error - I am receiving a ValueError: need more > than 1 value to unpack > > I have pasted the code and full error below. Any help will be greatly > appreciated. > > > > > import numpy as np > > import netCDF4 > > > def naive_fast(latvar,lonvar,lat0,lon0): > ># Read latitude and longitude from file into numpy arrays > >latvals = latvar[:] > >lonvals = lonvar[:] > >ny,nx = latvals.shape > >dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 > >minindex_flattened = dist_sq.argmin() # 1D index of min element > >iy_min,ix_min = np.unravel_index(minindex_flattened, latvals.shape) > >return iy_min,ix_min > > filename = "/Users/T_SFC.nc" > > ncfile = netCDF4.Dataset(filename, 'r') > > latvar = ncfile.variables['latitude'] > > lonvar = ncfile.variables['longitude'] > > > iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) > > print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] > > ncfile.close() > > > > > > --- > ValueErrorTraceback (most recent call last) > /Applications/Canopy.app/appdata/canopy-1.3.0.1715.macosx-x86_64/Canopy.app/Contents/lib/python2.7/site-packages/IPython/utils/py3compat.pyc > in execfile(fname, *where) > 202 else: > 203 filename = fname > --> 204 __builtin__.execfile(filename, *where) > > /Users/latlon_to_closestgrid.py in () > 22 lonvar = ncfile.variables['longitude'] > 23 > ---> 24 iy,ix = naive_fast(latvar, lonvar, -38.009, 146.438) > 25 print 'Closest lat lon:', latvar[iy,ix], lonvar[iy,ix] > 26 ncfile.close() > > /Users/latlon_to_closestgrid.py in naive_fast(latvar, lonvar, lat0, lon0) > 12 latvals = latvar[:] > 13 lonvals = lonvar[:] > ---> 14 ny,nx = latvals.shape > 15 dist_sq = (latvals-lat0)**2 + (lonvals-lon0)**2 > 16 minindex_flattened = dist_sq.argmin() # 1D index of min > element > > ValueError: need more than 1 value to unpack > > > It seems that latvals and lonvals should be a 2D array and you are providing just a 1D array. Maybe you could use numpy.meshgrid [1] to get 2D inputs from 1D arrays. [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.meshgrid.html > > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fwd: Numpy for data manipulation
2015-10-02 9:38 GMT+02:00 Alex Rogozhnikov : > I would suggest >> >> %matplotlib notebook >> >> It will still have to a nice png, but you get an interactive figure when >> it is live. >> > > Amazing, thanks. I was using mpld3 for this. > (for some strange reason I need to put %matplotlib notebook before each > plot) > You should create a figure before each plot instead of putthon %matplotlib notebook plt.figure() > > The recommendation of inverting a permutation by argsort'ing it, while it >> works, is suboptimal, as it takes O(n log(n)) time, and you can do it in >> linear time: >> > Actually, there is (later in post) a linear solution using bincount, but > your code is definitely better. Thanks! > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fwd: Numpy for data manipulation
2015-10-02 9:48 GMT+02:00 Kiko : > > > 2015-10-02 9:38 GMT+02:00 Alex Rogozhnikov : > >> I would suggest >>> >>> %matplotlib notebook >>> >>> It will still have to a nice png, but you get an interactive figure when >>> it is live. >>> >> >> Amazing, thanks. I was using mpld3 for this. >> (for some strange reason I need to put %matplotlib notebook before each >> plot) >> > > You should create a figure before each plot instead of putthon %matplotlib > notebook > plt.figure() > > putthon == putting > > >> >> The recommendation of inverting a permutation by argsort'ing it, while it >>> works, is suboptimal, as it takes O(n log(n)) time, and you can do it in >>> linear time: >>> >> Actually, there is (later in post) a linear solution using bincount, but >> your code is definitely better. Thanks! >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: scipy 0.17.0 release
is it python3.5 compatible? your message and github don't say the same. 2016-01-23 19:12 GMT+01:00, Charles R Harris : > > > Congratulations. > > Chuck > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: scipy 0.17.0 release
BTW, congratulations and thanks for the hard work 2016-01-23 20:12 GMT+01:00, Kiko : > is it python3.5 compatible? your message and github don't say the same. > > 2016-01-23 19:12 GMT+01:00, Charles R Harris : >> >> >> Congratulations. >> >> Chuck >> > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyData Madrid
2016-02-20 17:58 GMT+01:00 Ralf Gommers : > > > On Wed, Feb 17, 2016 at 9:46 PM, Sebastian Berg < > sebast...@sipsolutions.net> wrote: > >> On Mi, 2016-02-17 at 20:59 +0100, Jaime Fernández del Río wrote: >> > Hi all, >> > >> > I just found out there is a PyData Madrid happening in early April, >> > and it would feel wrong not to go, it being my hometown and all. >> > >> > Aside from the usual "Who else is going? We should meet!" I was also >> > thinking of submitting a proposal for a talk. My idea was to put >> > something together on "The future of NumPy indexing" and use it as an >> > opportunity to raise awareness and hopefully gather feedback from >> > users on the proposed changes, in sort of a "if the mountain won't >> > come to Muhammad" type of thing. >> > >> >> I guess you do know my last name means mountain in german? But if >> Muhammed might come, I should really improve my arabic ;). >> >> In any case sounds good to me if you like to do it, I don't think I >> will go, though it sounds nice. >> > > Sounds like a good idea to me too. I like both the concrete topic, as well > as just having a talk on Numpy at a PyData conference. In general there are > too few (if any) talks on Numpy and other core libraries at PyData and > Scipy confs I think. > +1. It would be great a numpy talk from a core developer. BTW, C4P closes tomorrow!!! Jaime, if you come to Madrid you know you have some beers waiting for you. Disclaimer, I'm one of co-organizers of the PyData Madrid. Best. > Ralf > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PyData Madrid
2016-02-20 20:13 GMT+01:00 David Cournapeau : > > > On Sat, Feb 20, 2016 at 5:26 PM, Kiko wrote: > >> >> >> 2016-02-20 17:58 GMT+01:00 Ralf Gommers : >> >>> >>> >>> On Wed, Feb 17, 2016 at 9:46 PM, Sebastian Berg < >>> sebast...@sipsolutions.net> wrote: >>> >>>> On Mi, 2016-02-17 at 20:59 +0100, Jaime Fernández del Río wrote: >>>> > Hi all, >>>> > >>>> > I just found out there is a PyData Madrid happening in early April, >>>> > and it would feel wrong not to go, it being my hometown and all. >>>> > >>>> > Aside from the usual "Who else is going? We should meet!" I was also >>>> > thinking of submitting a proposal for a talk. My idea was to put >>>> > something together on "The future of NumPy indexing" and use it as an >>>> > opportunity to raise awareness and hopefully gather feedback from >>>> > users on the proposed changes, in sort of a "if the mountain won't >>>> > come to Muhammad" type of thing. >>>> > >>>> >>>> I guess you do know my last name means mountain in german? But if >>>> Muhammed might come, I should really improve my arabic ;). >>>> >>>> In any case sounds good to me if you like to do it, I don't think I >>>> will go, though it sounds nice. >>>> >>> >>> Sounds like a good idea to me too. I like both the concrete topic, as >>> well as just having a talk on Numpy at a PyData conference. In general >>> there are too few (if any) talks on Numpy and other core libraries at >>> PyData and Scipy confs I think. >>> >> >> +1. >> >> It would be great a numpy talk from a core developer. BTW, C4P closes >> tomorrow!!! >> >> Jaime, if you come to Madrid you know you have some beers waiting for you. >> >> Disclaimer, I'm one of co-organizers of the PyData Madrid. >> > > Since when does one need disclaimer when offering beers ? That would make > for a dangerous precedent :) > The disclaimer is not for the beers :-P The beers sentence should be a "P.D.:" > > David > >> >> Best. >> >> >>> Ralf >>> >>> >>> ___ >>> NumPy-Discussion mailing list >>> NumPy-Discussion@scipy.org >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] From Python to Numpy
2016-12-22 17:44 GMT+01:00 Nicolas P. Rougier : > > Dear all, > > I've just put online a (kind of) book on Numpy and more specifically about > vectorization methods. It's not yet finished, has not been reviewed and > it's a bit rough around the edges. But I think there are some material that > can be interesting. I'm specifically happy with the boids example that show > a nice combination of numpy and matplotlib strengths. > > Book is online at: http://www.labri.fr/perso/ > nrougier/from-python-to-numpy/ > Sources are available at: https://github.com/rougier/from-python-to-numpy > > > Comments/questions/fixes/ideas are of course welcome. > Wow!!! Beautiful. Thanks for sharing. > > > Nicolas > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fortran order in recarray.
2017-02-22 16:23 GMT+01:00 Alex Rogozhnikov : > Hi Francesc, > thanks a lot for you reply and for your impressive job on bcolz! > > Bcolz seems to make stress on compression, which is not of much interest > for me, but the *ctable*, and chunked operations look very appropriate to > me now. (Of course, I'll need to test it much before I can say this for > sure, that's current impression). > > The strongest concern with bcolz so far is that it seems to be completely > non-trivial to install on windows systems, while pip provides binaries for > most (or all?) OS for numpy. > I didn't build pip binary wheels myself, but is it hard / impossible to > cook pip-installabel binaries? > http://www.lfd.uci.edu/~gohlke/pythonlibs/#bcolz Check if the link solves the issue with installing. > > You can change shapes of numpy arrays, but that usually involves copies > of the whole container. > > sure, but this is ok for me, as I plan to organize column editing in > 'batches', so this should require seldom copying. > It would be nice to see an example to understand how deep I need to go > inside numpy. > > Cheers, > Alex. > > > > > 22 февр. 2017 г., в 17:03, Francesc Alted написал(а): > > Hi Alex, > > 2017-02-22 12:45 GMT+01:00 Alex Rogozhnikov : > >> Hi Nathaniel, >> >> >> pandas >> >> >> yup, the idea was to have minimal pandas.DataFrame-like storage (which I >> was using for a long time), >> but without irritating problems with its row indexing and some other >> problems like interaction with matplotlib. >> >> A dict of arrays? >> >> >> that's what I've started from and implemented, but at some point I >> decided that I'm reinventing the wheel and numpy has something already. In >> principle, I can ignore this 'column-oriented' storage requirement, but >> potentially it may turn out to be quite slow-ish if dtype's size is large. >> >> Suggestions are welcome. >> > > You may want to try bcolz: > > https://github.com/Blosc/bcolz > > bcolz is a columnar storage, basically as you require, but data is > compressed by default even when stored in-memory (although you can disable > compression if you want to). > > > >> >> Another strange question: >> in general, it is considered that once numpy.array is created, it's shape >> not changed. >> But if i want to keep the same recarray and change it's dtype and/or >> shape, is there a way to do this? >> > > You can change shapes of numpy arrays, but that usually involves copies > of the whole container. With bcolz you can change length and add/del > columns without copies. If your containers are large, it is better to > inform bcolz on its final estimated size. See: > > http://bcolz.blosc.org/en/latest/opt-tips.html > > Francesc > > >> >> Thanks, >> Alex. >> >> >> >> 22 февр. 2017 г., в 3:53, Nathaniel Smith написал(а): >> >> On Feb 21, 2017 3:24 PM, "Alex Rogozhnikov" >> wrote: >> >> Ah, got it. Thanks, Chris! >> I thought recarray can be only one-dimensional (like tables with named >> columns). >> >> Maybe it's better to ask directly what I was looking for: >> something that works like a table with named columns (but no labelling >> for rows), and keeps data (of different dtypes) in a column-by-column way >> (and this is numpy, not pandas). >> >> Is there such a magic thing? >> >> >> Well, that's what pandas is for... >> >> A dict of arrays? >> >> -n >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > > -- > Francesc Alted > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
