I would be in agree. thanks! I use gawk to separate the file into many files by year, then it would be easier to handle. anyway, it's not a good practice to produce such huge line txt files....
Chao 2012/3/20 David Froger <[email protected]> > Hi, > > I think writing a Python script that convert your txt file to one netcdf > file, > reading the txt file one line at a time, and then use the netcdf file > normally > would be a good solution! > > Best, > David > > Excerpts from Chao YUE's message of mar. mars 20 13:33:56 +0100 2012: > > Dear all, > > > > I received a file from others which contains ~30 million lines and in > size > > of ~500M. > > I try read it with numpy.genfromtxt in ipython interactive mode. Then > > ipython crashed. > > The data contains lat,lon,var1,year, the year ranges from 1001 to 2006. > > Finally I want to write the > > data to netcdf for separate years and feed them into the model. I guess I > > need a better way to do this? > > anyone would be any idea is highly appreciated. > > > > > > lon,lat,year,area_burned > > -180.0,65.0,1001,0 > > -180.0,65.0,1002,0 > > -180.0,65.0,1003,0 > > -180.0,65.0,1004,0 > > -180.0,65.0,1005,0 > > -180.0,65.0,1006,0 > > -180.0,65.0,1007,0 > > > > thanks and cheers, > > > > Chao > > > _______________________________________________ > NumPy-Discussion mailing list > [email protected] > http://mail.scipy.org/mailman/listinfo/numpy-discussion > -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************
_______________________________________________ NumPy-Discussion mailing list [email protected] http://mail.scipy.org/mailman/listinfo/numpy-discussion
